Tuesday, July 21, 2009

On Relying on Self-Report: Happiness and Charity

To get published in a top venue in sociology or social or personality psychology, one must be careful about many things -- but not about the accuracy of self-report as a measure of behavior or personality. Concerns about the accuracy of self-report tend to receive merely a token nod, after which they are completely ignored. This drives me nuts.

(Before I go further, let me emphasize that the problem here -- what I see as a problem -- is not universal: Some social psychologists -- Timothy Wilson, Oliver John, and Simine Vazire for example -- are appropriately wary of self-report.)

Although the problem is by no means confined to popular books, two popular books have been irking me acutely in this regard: The How of Happiness, by my UC Riverside colleague Sonja Lyubomirsky, and Who Really Cares, by Arthur Brooks (who has a named chair in Business and Government Policy at Syracuse).

The typical -- but not universal -- methodology in work by Lyubomirsky and those she cites is this: (A1.) Ask some people how happy (or satisfied, etc.) they are. (A2.) Try some intervention. (A3.) Ask them again how happy they are. Or: (B1.) Randomly assign some people to two or three groups, one of which receives the key intervention. (B2.) Ask the people in the different groups how happy they are. If people report greater happiness in A3 than in A1, conclude that the intervention increases happiness. If people in the intervention group report greater happiness in B2 than people in the other groups, likewise conclude that the intervention increases happiness.

This makes me pull out my hair. (Sorry, Sonja!) What is clear is that, in a context in which people know they are being studied, the intervention increases reports of happiness. Whether it actually increases happiness is a completely different matter. If the intervention is obviously intended to increase happiness, participants may well report more happiness post-intervention simply to conform to their own expectations, or because they endorse a theory on which the intervention should increase happiness, or because they've invested time in the intervention procedure and they'd prefer not to think of their time as wasted, or for any of a number of other reasons. Participants might think something like, "I reported a happiness level of 3 before, and now that I've done this intervention I should report 4" -- not necessarily in so many words.

As Dan Haybron has emphasized, the vast majority of the U.S. population describe themselves as happy (despite our high rate of depression and anger problems), and self-reports of happiness are probably driven less by accurate perception of one's level of happiness than by factors like the need to see and to portray oneself as a happy person (otherwise, isn't one something of a failure?). My own background assumption, in looking at people's self-reports of happiness, life-satisfaction, and the like, is that those reports are driven primarily by the need to perceive oneself a certain way, by image management, by contextual factors, by one's own theories of happiness, and by pressure to conform to perceived experimenter expectations. Perhaps there's a little something real underneath, too -- but not nearly enough, I think, to justify conclusions about the positive effects of interventions from facts about differences in self-report.

In Who Really Cares? Brooks aims to determine what sorts of people give the most to charity. Brooks bases his conclusions almost (but not quite) entirely on self-reports of charitable giving in large survey studies. His main finding is that self-described political conservatives report giving more to charity (even excluding religious charities) than do self-described political liberals. What he concludes -- as though this were unproblematically the same thing -- is that conservatives give more to charity than do liberals. Now maybe they do; it wouldn't be entirely surprising, and he has a little bit of non-self-report evidence that seems to support that conclusion (though how assiduously he looked for counterevidence is another question). But I doubt that people have any especially accurate sense of how much they really give to charity (even after filling out IRS forms, for the minority who itemize charitable deductions), and even if they did have such a sense I doubt that would be accurately reflected in self-report on survey studies.

As with happiness, I suspect self-reports of charitable donation are driven at least as much by the need to perceive oneself, and to have others perceive one, a particular way as by real rates of charitable giving. Rather than assuming, as Brooks seems to, that political conservatives and political liberals are equally subject to such distortional demands in their self-reports and thus attributing differences in self-reported charity to actual differences in giving, it seems to me just as justified -- that is to say, hardly justified at all -- to assume that the real rates of charitable giving are the same and thus attribute differences in reported charity to differences in the degree of distortion in the self-descriptive statements of political conservatives and political liberals.

Underneath sociologists' and social and personality psychologists' tendency to ignore the sources of distortion in self-report is this, I suspect: It's hard to get accurate, real-life measures of things like happiness and overall charitable giving. Such real-life measures will almost always themselves be only flawed and partial. In the face of an array of flawed options, it's tempting to choose the easiest of those options. Both the individual researcher and the research community as a whole then become invested in downplaying the shortcomings of the selected methods.

15 comments:

Unknown said...

Hi Eric - I enjoyed your post very much, and no apologies needed for constructive criticism! You obviously discuss a much larger and more complicated question than a brief comment can even touch on, but I wanted to bring up a couple issues: (1) in most of our intervention studies, we work very hard to include appropriate control conditions in which participants are led to believe that they are also doing something worthwhile to increase happiness (e.g., boosting their organizational and time management skills), yet those controls do not report increases in happiness -- a finding that your critique does not address; (2) I agree that sometimes people say they are happy because they think that's what's expected or socially desirable, etc.; however, I do not think that that happens most of the time; when someone truly believes that they are happy or have become happier as a result of an intervention, I (not being a philosopher!) do not split hairs about whether that means that they are REALLY happy or just BELIEVE that they are happy; they just are; (3) even psychologists who discuss the cons of self-report use it all the time in their research (just skim the papers). I agree that we should be more thoughtful about self-reports, but I also think that there are certain phenomena that psychologists study that are by their very nature subjective. First, you can have a disease and not know it, but you can't be happy and not know it. Second, no one else (e.g., peers, spouses, clinical psychologists) can tell you how happy you really are. Having said all this, I think there are some really fascinating philosophical issues to consider and I hope to do so in the Garrison Meetings in NY this August (where Haybron and representatives from several disciplines will meet to discuss well-being).

Best,

--Sonja (and bonjour de Paris!)

Jonathan Jenkins Ichikawa said...

Totally agree. This bothers me too.

Brad C said...

Hi Eric,

This reminds me of one of my favorite Hume quotes, which is interestingly suggestive of the limits of experimental philosophy:

"Moral philosophy has, indeed this peculiar disadvantage, which is not found in natural, that in collecting its experiments, it cannot make them purposely, with premeditation, and after such a manner as to satisfy itself concerning every particular difficulty which may arise. When I am at a loss to know the effects of one body upon another in any situation, I need only put them in that situation, and observe what results from it. But should I endeavour to clear up after the same manner any doubt in moral philosophy, by placing myself in the same case with that which I consider, ‘tis evident this reflection and premeditation would so disturb the operation of my natural principles, as must render it impossible to form any just conclusion from the phaenomenon. We must therefore glean up our experiments in this science from a cautious observation of human life, and take them as they appear in the common course of the world, by men’s behavior in company, in affairs, and in their pleasures."

Eddy Nahmias said...

As often happens in discussions of happiness, Eric and Sonja seem to be employing two different conceptions of happiness. Sonja writes, "you can't be happy and not know it," which is equivalent to: if you are happy, then you know it (clap your hands!!). And this claim is even stronger than: if you (honestly) report you are happy, then you are happy. These claims suggest a conception of happiness that is defined subjectively. To be happy is to think you are happy. And if that's the conception in play, then of course, self-report is an appropriate way to measure happiness (discussions of pain and measuring pain often take this form).

There are two ways Eric may be disagreeing with this conception of happiness. First, even on a subjective conception of happiness, it may be false that people can accurately *report* what they actually feel. There may be various reasons for this--perhaps Eric can say more about why he thinks this is the case.

Second, one may conceive of happiness as having some objective components (more like Aristotle's idea of flourishing). On this view, self-report could often be inaccurate. People high on crack may say they are really happy but they are not flourishing.

I'd like to see some experimental work done on how ordinary people conceive of happiness. I don't think they think of happiness as entirely subjective--e.g., I bet they would disagree with statements like, "If a drug addict high on meth says that he is happy, then he is happy."

Matthew Pianalto said...

Eddy is more or less right. Dan Haybron's book (and his other work) is a good place to start on this, and I think it also has the virtue of bridging the interests between philosophers and psychologists and social scientists here. For worries about self-reporting, one should look at Dan's account of "affective ignorance."

I don't have the references handy, but I recall reading some things (one piece was in Social Indicators Research, I believe) on "impression management," which is, roughly, saying what you think you ought to say. The study wasn't about happiness specifically, but provided some general support to the idea that worries about impression management might be overblown. It is true, if I recall correctly, that impression management is more pronounced during in-person interviews than, say, in mailed/written surveys. There are, I think, ways to detect impression management, and so one could use such methods to "weed out" people who can't be trusted to be sincere from one's study.

At the same time, I'm sympathetic with Haybron's worry that even if we are ignorant of our own (roughly) mood (etc.) some of the time, this can be problematic for such studies. I'm not sure, however, that the most careful researchers do downplay the limitations. A more significant problem might be how these limitations drop from view in more popular deliveries of the data (e.g. in the media).

(OTOH, those who want a more "objective" way of studying happiness, say, by doing fMRI's, might want to take a look at one of Fred Feldman's papers where here takes Richard Layard to task for claiming that neuroscientists have "shown happiness to exist" in the brain. Feldman seems pretty confident that asking people how happy they are is likely to be as reliable as anything else. Last I checked, it was on Feldman's website.)

dan haybron said...

Eddy raises an excellent question about the folk concept of happiness, which I've seen remarkably little about. Sven Nyholm is doing some interesting work that might offer some support for Eddy's conjecture.

I've done some surveys over the years using an experience-machine-type vignette, which I keep promising to post on the xphi blog (soon! though some results are in my book). Overwhelmingly, students say the deceived person is happy, yet a majority deny that he is doing well, flourishing, fortunate, etc. This suggests they treat 'happy' as a psychological term.

Interestingly, though, half or more students also say the person has a "happy life"--suggesting a "well-being" reading of that locution. That fits my intuitions: the EM user can be happy, but does not lead a happy life.

I think Eric is using 'happiness' in the same psychological sense as the psychologists, but doubting that self-reports of that psychology are reliable. (I don't think most of us know how happy we are, or how pleasant our lives are, or even, often, how we feel right now, except very crudely--like not in agony.) I agree, though I also agree with Sonja that the problems needn't be fatal to the research (as in teh controlled studies she cites). My guess is that, where norms don't systematically differ (eg, USA v France) self-reports correlate more or less decently w emotional state--happier people tend to report greater happiness. (However, some important aspects of our emotional lives, like stress, probably get short shrift.)

While self-reports will always be problematical, I think it would be helpful to shift away from self-reports of "happiness", which has so many different meanings--eg, some will take it as a question about life satisfaction, others about emotional fulfillment, etc. Better just to decide what exactly we want to measure--LS, emotional state--and measure that. People probably aren't very reliable judges of how stressed or irritable or depressed (etc) they are, but I think we'll get better data about their psychic well-being if we focus directly on those variables (or on life satisfaction-related variables, if that's what you think happiness is) than if we simply ask them how happy they are.

I suspect that one reason self-reports of "happiness" correlate so well with other things you'd expect them to is that, with happiness, pretty much everything correlates with everything else, so it isn't hard to come up with measures that have attractive psychometric properties. I'm looking forward to chatting with Sonja about this stuff in Garrison!

Dan

ps--thanks matt--your comment came in just as i was failing to post this one!

dan haybron said...

oops--I meant to say in the 3rd para that students tend to *deny* that the EM user had a happy life.

Eric Schwitzgebel said...

Thanks for all the thoughtful comments, folks!

Sonja: I agree that the controls you mention help allay some of the concerns, and those are the studies that seem a little more persuasive to me. However, clearly many studies do not have such controls; and furthermore such controls, while ruling out some of the alternative explanations still leave other possible explanations open (e.g., maybe people don't really buy the claim that improving organizational skills by the exercises offered really would increase happiness; maybe people required to do something many times feel more annoyed at the experiment than those who need to do it only a few times and so are less inclined to please the experimenter; etc.). The main problem, it seems to me, is that subjective report is so loosely tied to actual life experience, that there will always be multiple possible explanations. What we really need is convergent evidence from a variety of sources a substantial bulk of which don't require self-report.

This leads to the point raised by...

Sonja, Eddy, and Dan about possible conflicting definitions of happiness. Here's my take: On the one hand, I think my take on happiness isn't *too* far from the folk take or many psychologists' take: I think of "happiness" as a mushy conglomorate of goods, heavily weighted toward the psychological and, within the psychological, heavily weighted toward the affective. (Maybe I'll need a cleaner definition one day, but I haven't felt the need of it yet.) On the other hand, I clearly differ from Sonja to the extent she takes belief that one is happy as *criterial* for happiness. I remember discussing this with her briefly last fall. I think I got her to concede that the suicidally depressed person loaded with visible negative affect who nonetheless sincerely protests that she is happy, really, is not actually happy. Granting that, belief cannot be criterial for happiness. Then the question is the empirical relationship between belief and possession of the goods (esp. psychological-affective) constitutive of happiness; and here I'm skeptical. Note at a minimum: Reports of *general* levels of happiness seem to require some sort of integration or averaging over time, and there's little reason to believe people would be especially reliable in such matters.

Eric Schwitzgebel said...

Brad: Thanks for reminding us of that nice quote from Hume! From context it seems evident that he is using "moral" there not in the narrow, contemporary sense, but rather in the broader, now obsolete sense of "concerning human nature and the mind".

Eric Schwitzgebel said...

Eddy: That's nicely put (and see my response above). I also agree that it would be interesting to see more work on folk conceptions of happiness.

Matthew: Thanks for the references. I know only a little of the literature on impression management, but I'm inclined for various reasons to think it's still very likely to be problematic, even in anonymous surveys. In my own recent anonymous survey, for example, it seems pretty clear that there's either a lot of self-deception or a lot of impression management going on, to judge (for example) from professors' evidently grossly exaggerated self-descriptions of degree of responsiveness to student emails. I'm not sure there is a clear line between self-deception and impression management: We tend to half-believe our own lies.

Eric Schwitzgebel said...

P.S. Matt: On Feldman vs. Layard, let me just re-emphasize that I agree that simplistic objective measures won't suffice either, except possibly a diverse range of them in combination.

Dan (in addition to the response above): Those are interesting results about the experience-machine thought experiment. I hope you have the chance to write them up more formally before long. I agree with you on most of your other points, except to my ear you sound, still, a bit too optimistic about the correlation of reports and actual happiness. If group X reports more happiness than group Y, my first hypothesis would be that there's some social-reporting factor driving the difference, unless there's excellent *independent* reason to think that there are substantial happiness-influencing life differences between the groups.

Matthew Pianalto said...

Eric: on your last comment, check this out. (This is partly tongue in cheek, but only partly...)

Eric Schwitzgebel said...

Cute! Thanks for the link. Of course, I don't believe a word of it. (Not that I positively *disbelieve* it either.)

Neuroskeptic said...

Yeah, this is absolutely right.

It's also a problem in psychiatry when it comes to measuring depression, anxiety etc.

whateve3r said...

Great post. As long as you're finding biases in happiness research, how about talking about the types of happiness that are researched. Positive emotions come in many flavors, but the ones that are studied seem to be those that psychology professors are especially prone to. Flow, for example, is going to be a feeling that academics probably feel more often, and more strongly, than other people. Otherwise they wouldn't go into a field which demand so much self-motivation. Emotions that the typical academic feels less often, such as exuberance, joy, elation, and feelings from social events or "letting loose", seem to be rarely studied, even though, from all outward signs (smiles, laughter, screams of happiness), these emotions seem more in line with what strong happiness seems to be.