How Reliable Are Happiness Self-Reports
That’s “reliable” in the technical measurement sense of “repeatability” or “consistency.” Alan Krueger and David Schkade are on the case with a new NBER paper, “The Reliability of Subjective Well-Being Measures.”
ABSTRACT
This paper studies the test-retest reliability of a standard self-reported life satisfaction measure and of affect measures collected from a diary method. The sample consists of 229 women who were interviewed on Thursdays, two weeks apart, in Spring 2005. The correlation of net affect (i.e., duration-weighted positive feelings less negative feelings) measured two weeks apart is 0.64, which is slightly higher than the correlation of life satisfaction (r=0.59). Correlations between income, net affect and life satisfaction are presented, and adjusted for attenuation bias due to measurement error. Life satisfaction is found to correlate much more strongly with income than does net affect. Components of affect that are more person-specific are found to have a higher test-retest reliability than components of affect that are more specific to the particular situation. While reliability figures for subjective well-being measures are lower than those typically found for education, income and many other microeconomic variables, they are probably sufficiently high to support much of the research that is currently being undertaken on subjective well-being, particularly in studies where group means are compared (e.g., across activities or demographic groups).
The passage in bold is not exactly a ringing endorsement, and definitely a call for caution, the implication being that the reliability of SWB measures are insufficient for some current research. I look forward to digging in deeper.
[Thanks to Tyler Cowen for the tip.]
7 Comments so far
Leave a reply
Will,
Has anybody else asked “where did this happiness survey methodology come from?” I’m asking and the answers I get point down some fascinating — perhaps scary — paths. Jean Converse, in her history of survey research in the USA, mentions an observation by Joseph Wood Krutch to the effect that happiness is something that cannot be measured. Her response to Krutch’s view:
“Perhaps not, but social scientists would soon be intent on trying to do so. [Paul] Lazarsfeld, in fact, had already put that subject, the measurement of personal happiness, on the research agenda.”
Where did Lazarsfeld get his impetus for putting the measurement of happiness on the research agenda? From his 1930s research on unemployment in Marienthal, Austria and from the Max Weber/Verein fur Sozialpolitik study of the psychophysics of industrial labor that informed the methodology of the Marienthal research. Two sources that are invaluable for tracking down the emergence of the happiness methodology are Lazarsfeld and Oberschall’s “Max Weber and Emprical Social Research” and Robert Brain’s “The Ontology of the Questionnaire — Max Weber on Measurement and Mass Investigation.
Will,
I was just at the Central American Philosophical Association conference, and Robert Woolfolk (a pscyhologist/pscyhotherapist) gave a talk that hit on some general problems he has with the self-report methodology:
1. Fallibility of self-knowledge
2. Impression management (wanting to “pass” the survey “test”)
3. “The unconscious rules”
I found myself wondering whether and to what extent groups that are given happiness surveys have also been “tested” to see just how much (or what percentage of the sample) are heavy impression managers, which would tend to skew the data on how happy people are upward. Maybe you know of some discussions of this problem?
Matt,
That’s sounds fascinating, but I don’t know of any good discussions offhand.
Re. point number 2, Eugene Osin (PhD student at Moscow State University) is doing some work on self-deception / social desirability bias in relation to various happiness self-report scales. He had a paper at the European Positive Psychology Conference last year.
I’ve just dug out the abstract. To summarise, he found that standard measures of self-deception did not correlate significantly with happiness or life satisfaction. However, they correlated positively with various measures of autonomy, resilience and purpose-in-life and negatively with some of Deci & Ryan’s causality orientation scales.
Sam
I’ve found a lot of interesing research studies on happiness over at www.notsalmon.com — the author Karen Salmansohn also hosts a sirius radio show about happiness called BE HAPPY DAMMIT which often cites different research studies on happiness you might find interesting to check out.
Follow-up: I found the following paper: “Happiness and Self-Deception: An Old Question Examined by a New Measure of Subjective Well-Being” by James Hagedorn in Social Indicators Research 38 (1996). Hagedorn reports:
“Self-Deception correlated significantly with Overall Life [Satisfaction]…while Impression Management did not….it is not others that are meant to be deceived by high life satisfaction ratings, but rather oneself. Unanswered is the question of whether or not self-deception is a healthy part of well-being.” (158)
There’s some fodder…
Will,
Can you elaborate on, “the implication being that the reliability of SWB measures are insufficient for some current research?”
With what sorts of current research do you think folks are drawing spurious conclusions, and where do you believe we are on safe ground? And how exactly does this relate to test-retest reliability, when compared to other factors such as education?
Thanks!
Matt