CHAPTER1 Basic Concepts in Psychological Measurement Contents 1.1. Some Simple Statistical Ideas 2 1.1.1 Levels of measurement 2 1.1.2 Standard scores 3 1.1.3 Correlation coefficients 5 1.2. Assessing Quality of Measurement: Reliability and Validity 9 1.2.1 Reliability 9 1.2.1.1 Internal-consistency reliability 10 1.2.1.2 Interrater (interobserver) reliability 13 1.2.1.3 Test-retest reliability 14 1.2.2 Validity 16 1.2.2.1 Content validity 16 1.2.2.2 Construct validity: convergent and discriminant 17 1.2.2.3 Criterion validity 18 1.3. Methods of Measurement: Self- and Observer Reports, Direct Observations, Biodata 19 1.3.1 Self-reports 19 1.3.2 Observer reports 20 1.3.3 Direct observations 21 1.3.4 Biodata (life outcome data) 22 1.3.5 Comparing the methods of measurement 22 1.4. Summary and Conclusions 24 Before we can really begin to understand personality, we need to figure out how to measure it. And before we can measure personality, it would be useful to have some common terms for describing our measurements. In this chapter, we will introduce some basic concepts that allow us to describe psychological measurements. By using these concepts, we will have some quick and simple ways of understanding the results of personality research. For example, if a researcher reports that women have a higher level than do men of some personality characteristic, you would probably want to know how much higher. Or, if a researcher reports that a given personality characteristic is related to enjoyment of a particular kind of music, you would probably want to know how much they are related. Also in this chapter, we will consider the basic ways of evaluating whether or not our measurements are accurate. Whenever we try to measure a psychological Individual Differences and Personality Ó 2013 Elsevier Inc. ISBN 978-0-12-416009-5, http://dx.doi.org/10.1016/B978-0-12-416009-5.00001-3 All rights reserved. 1 2 Individual Differences and Personality characteristic, we need to make sure (a) that we really have measured some meaningful characteristic, and (b) that this characteristic really is the same one that we are trying to measure. Measuring psychological characteristics accurately can be tricky, so it is important to have some ways of expressing how well we have measured those characteristics. And finally, we will also explore in this chapter some of the methods that psychol- ogists use when measuring personality and related characteristics. As you will see later in this chapter, there exists a variety of methods, each of which has its advantages and disadvantages. 1.1. SOME SIMPLE STATISTICAL IDEAS 1.1.1 Levels of measurement One difficulty in psychological measurement, as opposed to measurement in other areas of science, is that there is usually not a meaningful “zero” level of a psychological trait. When astronomers describe a variable such as distance, it is easy to imagine what zero distance is. But when psychologists describe a characteristic such as intelligence or rebelliousness or irritability, it is difficult to imagine what a zero level would be. Even if someone has a score of zero on an intelligence test, it does not seem meaningful to say that the person has zero intelligencedpresumably, he or she would get a score higher than zero if the test were easier. Because there is no clear zero point, we cannot really say that one person is “three times” more rebellious or “50%” less irritable than another, in the way that an astronomer might say that one planet has twice as much mass as another. That is, in psychology we are usually not able to describe ratios between people’s levels of a variable or a person’s absolute amount of a variable. But the lack of a true zero level of psychological characteristics does not mean that we cannot measure those characteristics. In fact, there are several different ways by which we could compare people’s levels of any given trait. One of these is simply to rank people: For example, we could measure people’s levels of ambition, and then record their positions relative to each other, such as 1st, 2nd, 3rd, ., 654th, .. This is a sensible approach, but it has some shortcomings. One difficulty is that the differences between the ranks are not always meaningful. For example, the person with the highest level of a trait in a given sample might be just slightly higher than the person with the 2nd highest level, but the person who is 2nd highest might be far, far ahead of the person in 3rd. This fact means that ranks are less than ideal for calculating statistics based on our measurements. For example, when we want to compute the average level of a trait, our computation is much more meaningful if the differences (or “intervals”) between the numbers always mean the same thing. So, the numbers provided by ranks are not as useful as we might like them to be. Basic Concepts in Psychological Measurement 3 In measuring people’s characteristics, therefore, psychologists would like to obtain scores that have meaningful differences between them (even though the ratios need not be meaningful). For example, if we are trying to measure the trait of “assertiveness,” we would like to be confident that a score of 60 really does mean a level of the trait that is halfway between the levels indicated by a 50 and a 70. Note that assertiveness is not really being measured in any particular “units,” and that it does not matter if the average score is 60, or 360, or 60, or whatever. The important thing, for the purpose of making meaningful comparisons among people and of being able to calculate statistics, is simply that equal differences, or intervals, between scores represent roughly equal differences in the level of the trait. For example, when psychologists measure intelligence using an “IQ” test, they would hope that the difference between an IQ of 110 and an IQ of 120 really does have the same meaning as the difference between an IQ of 130 and an IQ of 140. (Note again, by the way, that an IQ of 0 does not indicate zero intelligence; note also that the average IQ level has been set arbitrarily at 100, even though any other value could have been chosen as the average instead.) How do psychologists know if their measurements meet the requirement of having meaningful differences? The methods for testing this are beyond the scope of this textbook, but we can say here that most well-designed psychological measurements are close enough to this ideal to be useful for statistical analysis. 1.1.2 Standard scores It was mentioned before that psychological characteristics are not measured in any particular “units,” and that it does not matter how high or low the “scores” on a char- acteristic tend to be, as long as the differences between scores are meaningful. But differences in the numbers used for measuring variables might cause difficulties when we want to compare someone’s scores across two or more traits. For example, suppose that Bob has an IQ of 90 (where the average person’s IQ is 100) and that Bob also has a score of 60 on a “sociability” scale (which, let us say, has an average score of 50). At first glance, it seems that Bob’s IQ score (90) is higher than his sociability score (60), but in fact Bob is below the average on IQ and above the average on sociability. Therefore, we need some way to relate scores on one scale to scores on another scale, so that we can compare levels of one characteristic with levels of another, or to compare scores on the same charac- teristic as measured by different scales. Psychologists are able to make meaningful comparisons across different kinds of measurement scales by converting scores into standard scores. The first step in calculating a standard score is to take an individual’s score on a given scale, and then subtract the mean score (i.e., the average score) for the persons who have been measured. This difference between the individual’s score and the mean score tells us whether the person is above the average (if the difference is positive) or below the average (if the difference is negative). 4 Individual Differences and Personality But this is not the only step. If we merely subtract the mean score from the indi- vidual’s score, we still might not have a meaningful idea of how far above or below the average that person is. This is because different scales of measurement differ in terms of how “spread out” people’s scores are. For example, on a typical IQ test, about two-thirds of people are within 15 points of the average (i.e., between 85 and 115), and about 95% of people are within 30 points of the average (i.e., between 70 and 130). So, a person who has an IQ of 110 is above average, but not very far above average. But imagine that we have a different test, on which people’s scores are much more tightly bunched (say, two-thirds of people between 95 and 105, and 95% of people between 90 and 110). On this scale, a score of 110 would be very high. So, we need some way to compare scales that have different amounts of variability in people’s scores, as well as different average scores. In order to do this, psychologists use a second step, after having first subtracted the average score on a scale from the individual’s score on that scale. They then divide this BOX 1-1 The Normal Distribution The examples in the text are based on what is called a normal distribution of scores; when drawn as a graph, this produces the well-known bell-shaped curve (see Figure 1-1). For many physical and psychological characteristics, the distribution of scores is roughly normal: Most people have scores close to the average value, with relatively few people being far above or far below that average. Notice that a person whose score is equal to the mean will have a score that is higher than that of 50% of people. If a person’s score is one standard deviation above z-score –3.5 –3 –2.5 –2 –1.5 –1 –0.5 0 0.5 1 1.5 2 2.5 3 3.5 % of scores below <1 2 7 16 31 50 69 84 93 98 >99 Figure 1-1 The normal distribution of scores. the mean, then his or her score is higher than that of about 84% of people; if it is two standard deviations above, then it is higher than that of about 98% of people. Conversely, a score that is one standard deviation below the mean is higher than that of about 16% of people, and a score that is two standard deviations below is higher than that of about 2% of people. Basic Concepts in Psychological Measurement 5 difference by the standard deviation, a number that indicates how much variability there 1 is among people on a variable. For many psychological characteristics, about two- thirds of people are within one standard deviation above or below the mean, and about 95% of people are within two standard deviations above or below the mean. (For example, in the situation mentioned before for the typical IQ test, the standard devi- ation is 15.) The result of the preceding two stepsdfinding the difference between the individ- ual’s score and the average score, and then dividing this difference by the variability (standard deviation) of the scoresdis to give a universal or standard way of expressing people’s scores on a given characteristic, regardless of the original distribution of scores on that characteristic. These scores, known as standard scores, have two special prop- erties: First, the average score on a standard-score scale is exactly zero, and second, the standard deviation of a standard-score scale is exactly one. So, after we have calculated standard scores for our variables, we can meaningfully compare a person’s scores across different variables. This applies not only to different scales measuring the same variable (e.g., two different IQ test scales), but also to scales measuring different variables (e.g., an 2 IQ test scale and a sociability scale, or an “orderliness” scale and an “originality” scale). 1.1.3 Correlation coefficients The correlation coefficient, known by the symbol r, tells us how strongly two variables “go together”, or covary with each other. The values of the correlation coefficient can range from a maximum of +1 (indicating a perfect positive correlation between two variables) to a minimum of 1 (indicating a perfect negative correlation between two variables). A correlation of 0 means that the two variables are unrelated to each other. 1 The standard deviation is calculated by, first, finding the difference between each person’s score and the average score across all persons, then squaring each of these values, then adding up all of the squared values, then dividing this sum by the number of people who were measured, and then finding the square root of this result. Suppose that there are N individuals, and each of the individuals, i, has a score of Xi on variable X, with a mean score of m for those individuals. The formula for the standard deviation, s, will then be as follows: vffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi u u u"PN ðXi mÞ2# u t i¼l s ¼ N If we do not take the square root, then we have the square of the standard deviation of scores, which is called the 2 variance of scores, s . 2 Sometimes, standard scores are converted from their basic form, in which the mean is zero and the standard deviation is one, into some other form. As mentioned above, IQ scores are nowadays calculated as standard scores, but with the mean set to 100 and the standard deviation set to 15. Many other tests use a form of standard scores called T-scores, whereby the mean is set to 50 and the standard deviation to 10. Scores on the well-known SAT examination, used in admissions to higher education in the United States, were originally standard scores for which the mean was set to 500 and the standard deviation to 100. 6 Individual Differences and Personality The correlation coefficient tells us an important fact about how people’s score on the two variables are related: A difference of 1 standard deviation unit on one variable is associated with a difference of r standard deviation units on the other variable. Let us consider some examples that show what this means. Suppose that variables X and Y have a perfect positive correlation with each other (i.e., r ¼ 1): the higher variable X is, the higher variable Y must be, and vice versa. In this case, we know that a person’s level of variable X (expressed in z-score units) will be equal to his or her level of variable Y (expressed in z-score units). For example, a person who is 1 standard deviation above the mean on variable X must also be 1 standard deviation above the mean on variable Y. Likewise, a person who is 2 standard deviations below the mean on variable Y must also be 2 standard deviations below the mean on variable X. Figure 1-2 (panel a) shows a graph that depicts what a correlation of +1 looks like; notice that the dots (each of which represents a person’s score on the two variables) make a straight line from the lower left of the graph (indicating low levels of both variable X and variable Y) to the upper right of the graph (indicating high levels of both variable X and variable Y). Now suppose instead that variables X and Y have a perfect negative correlation with each other (i.e., r ¼ 1): the higher variable X is, the lower variable Y must be, and vice versa. In this case, we know that a person’s level of variable X (expressed in z-score units) will be equal in size but opposite in sign to his or her level of variable Y (expressed in z-score units). A person who is 1.5 standard deviations above the mean on variable X must be 1.5 standard deviations below the mean on variable Y. Likewise, a person who is 0.5 standard deviations above the mean on variable Y must be 0.5 standard deviations below the mean on variable X. Figure 1-2 (panel b) shows a graph that depicts what a correlation of 1 looks like; notice that the dots (each of which represents a person’s score on the two variables) make a straight line from the upper left of the graph (indicating high levels of Yand low levels of X) to the lower right of the graph (indicating low levels of Y and high levels of X). It is unusual for two variables to correlate perfectly with each other, either positively or negatively. In fact, even when we measure the same characteristic in two different ways, we usually find a correlation smaller than +1, due to various sources of error in measurement (as we will discuss later in this chapter, in the section on Reliability). When the correlation between two variables is not +1 or 1, then we do not know exactly what a person’s level of one variable will be, simply by knowing his or her level of the other variable. However, for a large group of people who have a given level of one variable, the correlation does let us know roughly what the average level of the other variable will be for those particular people. For example, if the correlation between variable X and variable Y is .50, and if we have some people who are 2 standard deviations above the mean on variable X, then their average level on variable Y will be about 1 standard deviation above the mean Basic Concepts in Psychological Measurement 7 (a) r = +1.00 (b) r = −1.00 (c) r = +.50 (d) r = −.50 (e) r = .00 Figure 1-2 Scatterplots showing scores on variable Y (vertical axis) and variable X (horizontal axis) having correlations of (a) +1.00, (b) 1.00, (c) +.50, (d) .50, and (e) .00. (because 2 .50 ¼ 1). Likewise, if the correlation between variable X and variable Y is .50, and if we have some people who are 2 standard deviations above the mean on variable X, then we know that their average level on variable Y will be about 1 standard deviations below the mean (because 2 .50 ¼ 1). To get a sense of what a correlation of a given size “means”, consider the remaining panels of Figure 1-2. Notice that in these panels, the association between X and Y is not perfect, but you can still notice a tendency for X and Y to go together positively (panel c) or negatively (panel d). Figure 1-2 (c) shows a correlation of about +.50, which is a moderately large positive correlation. As an example, this might be close to the correlation that you would find, for a group of adults, between body weight and weightlifting ability. (On average, heavier people can lift more weight than lighter people can, but there are still some light people who can lift a lot and some heavy people who cannot lift very much.) Figure 1-2 (d) shows a correlation of about .50, which is a moderately large negative correlation. This might be close to the correlation that you would find, for a group of adults, between body weight and distance running ability. (On average, 8 Individual Differences and Personality heavier people cannot sustain as fast a running pace as lighter people can, but there are still some heavy people who can run at a fast pace and some light people who cannot run at a fast pace.) Figure 1-2 (e) shows a correlation of .00, which means that the two variables are completely unrelated to each other. A zero correlation (r ¼ .00) means that the two variables are completely unrelated to each other. In this case, people’s levels of variable X do not go along in either direction with their levels of variable Y. No matter what is the level of variable X shown by a given person, your best guess for that person’s level of variable Y is simply the average level for the entire sample of persons (i.e., 0 standard deviations from the mean). In this kind of situation, we say that the two variables are perfectly uncorrelated. Such a result might happen if we were to examine, in a group of adults, the correlation between height and intelligence. On average, taller people are probably no more and no less intelligent than shorter people are. There is no strict rule as to what value of a correlation makes it a “small” (or “low”, “weak”) as opposed to a “large” (or “high”, “strong”) correlation. But as a rough guideline, correlations of between about .20 and +.20 are often considered small, correlations between about .20 and .40 and between about .20 and .40 are considered moderate in size, and correlations beyond .40 or beyond +.40 are considered large. (A correlation of, say, +.80 is very large, and usually would be found only when the same variable is being measured by two similar methods.) Sometimes people tend to downplay the importance of correlations that are not very large. But even a rather modest correlation can provide useful information. For example, suppose that a personality variable correlates .25 with some outcome variable (such as job performance or marital satisfaction). What would this mean? People with very high levels of this personality variable (say, 2 standard deviation units above the mean) would, on average, be about 0.5 standard deviation units above the mean on the outcome variable (because 2 .25 ¼ .50). Likewise, people with very low levels of this personality variable (say, 2 standard deviation units below the mean) would, on average, be about 0.5 standard deviation units below the mean on the outcome variable (because 2 .25 ¼ .50). Therefore, there would be about a 1 standard deviation unit difference in the outcome variable between people who are very high and people who are very low in the personality variable (because .50 e ( .50) ¼ 1.0). A difference of 1 standard deviation unit is fairly large, so the information provided by this rather modest correlation coefficient is important. Here is the formula for calculating the correlation, r, between two variables, x and y, where Zxi and Zyi are standard scores on those variables for each of N individuals, i: N P ZxiZyi i¼1 rxy ¼ N Basic Concepts in Psychological Measurement 9 The idea is that for each of the N individuals we have measured, we find the product of his or her standard scores on the two variables. Then we add together the products obtained from each individual, and we divide this total by N, the number of individuals we have measured. Notice that, if most people tend to have positive z-scores on both variables or negative z-scores on both variablesdrather than a positive z-score on one variable and a negative z-score on the otherdthen the products will be positive, and their sum will be a positive number, thereby producing a positive correlation. If instead many people have a positive z-score on one variable and a negative z-score on the other, then the products will be negative, and their sum will be a negative number, thereby 3 producing a negative correlation. 1.2. ASSESSING QUALITY OF MEASUREMENT: RELIABILITY AND VALIDITY The preceding sections have described some of the basic statistical concepts that are used by psychologists who measure people’s levels of various characteristics. But now we need to consider the question of how to assess the quality of those measurements: When we try to measure a characteristic in a sample of people, how do we know whether or not we have been successful? In other words, how can we know how accurately we have measured that characteristic? There are several aspects of measurement quality to be considered, but these are generally classified into two broad properties known as reliability and validity. 1.2.1 Reliability The reliability of a measurement is the extent to which it agrees with other measurements of the same characteristic. When there is good agreement between measurements, this tells us that they are assessing some real characteristic, rather than just being meaningless random numbers. It is important to evaluate the reliability of our measurements, because whenever we try to measure a characteristic, there is likely to be some random error in those measurements. There are several different ways in which reliability can be assessed, depending on the kind of random error that we consider. For example, if we measure a characteristic by 3 Differences between two groups (such as men and women) on a variable can be expressed in terms of standard deviation units. For example, if the standard deviation of height for men and for women is 8 centimetres, and if the average man is 12 centimeters taller than the average woman, then we can say that the difference in height between men and women equals 1.5 standard deviation units. This method of expressing the differences between groups is related to the correlation coefficient. If, for example, men and women differ by 1.5 standard deviation units on a variable, then this is equivalent to a correlation of .60 between that variable and a person’s sex; likewise, a difference of 0.5 standard deviation units is equivalent to a correlation of about .25 and a difference of 1.0 standard deviation units is equivalent to a correlation of about .45. (The formula for this conversion is beyond the scope of this textbook.)