Justin Gengler, Social and Economic Survey Research Institute, Qatar University
This is part of the MENA Politics Newsletter, Volume 2, Issue 2, Fall 2019. Download the PDF of this article here.
Introduction
The previous decade has seen a marked expansion in the number and sophistication of scientific opinion surveys being used to study society and politics in Arab countries. This expansion has shifted focus from merely procuring public opinion data in a region that is often difficult to penetrate, to assessing and addressing concerns about the quality of Arab survey data and the substantive inferences based on them. Such methodological work continues, and includes studies of how the survey sponsor impacts participation; the effect on responses when third parties are present during the interview; and especially the influence of observable interviewer attributes on the responses given in surveys (interviewer effects). Underlying this methodological agenda is a lingering concern that certain aspects of the survey-taking climate in the Arab world and in other parts of the Middle East and North Africa—in particular, a lack of political freedom and conservative social norms—may undermine the representativeness and/or reliability of surveys, either by biasing samples in favor of certain types of respondents or by causing respondents to edit their answers to sensitive or even nonsensitive survey questions.
Yet these same contextual concerns should also prompt wider reflection about the meaning of public opinion research in the Arab world and in other settings where social or political factors may create a potentially inhospitable survey climate. How do men and women in Arab countries perceive surveys and participation in survey research? More fundamentally, how do they understand and define the concept “public opinion” in an environment far removed from its Western origins? Finally, are Arab views of surveys and conceptualizations of public opinion unique, or are they shared by people in other regions with political and/or cultural similarities? These are questions that colleagues and I have sought to better understand over the course of a three-year project on survey attitudes in the Arab world supported by the Qatar National Research Fund.[i] Our findings so far, recently published in the British Journal of Political Science, have important implications for consumers and producers of public opinion data from the Middle East and North Africa.
Doubts about Arab Survey Research
As a political scientist living and collecting survey data in the Arab Gulf states for more than a decade, meta-questions surrounding the meaning of public opinion in the MENA region emerged early in my work and have remained a constant backdrop to substantive research into political behavior, group conflict and identity politics, and authoritarianism. This is due, perhaps in the first place, to my experience presenting results from Gulf surveys at international survey research conferences since around 2010. Consistently, such presentations elicit some or all of the following, mostly rhetorical questions from audience members accustomed to hearing about polls conducted in the Netherlands or Quebec:
- “Do people in Arab countries understand surveys well enough to participate and give proper responses?”
- “Will a person even open the door for a stranger (interviewer) in those countries?”
- “How can you expect citizens in a non-democratic society to give honest answers to political or other sensitive questions in a survey?”
- “There is no such thing as public opinion in authoritarian Arab countries, so aren’t these survey data mostly meaningless?”
Even if overly exaggerated and sweeping, these reactions represent valid concerns about how specific aspects of the Arab survey climate might introduce nonresponse and response error—the same sort of doubts that have prompted extant methodological studies. Lack of public familiarity with surveys may depress response rates or make respondents less likely to follow survey instructions. Concerns over the privacy implications of surveys, including worry about possible government surveillance, may also lead to unit or item nonresponse error as well as socially desirable reporting. Additionally, it is possible that the nature of responses given in Arab opinion surveys is influenced by the way that individuals understand the idea of public opinion and that the modal Arab understanding is not equivalent to the conceptualization prevailing in the West.
Still, my anecdotal experience conducting interviews in non-democratic and socially conservative Arab Gulf states, particularly Bahrain and Qatar, gave a quite different impression. Not only do Gulf citizens, including residents of outlying rural villages, overwhelmingly recognize the purpose and modalities of surveys, but they often appear pleased to be asked their opinion on important issues and policies precisely because they are infrequently the subject of public debate. This includes during times of significant political tension in which concerns over possible state surveillance were very justified, as in Bahrain just prior to the country’s 2011 Shi‘a-led uprising. Observing my own family members’ complaints about telephone polls during U.S. election cycles, and brusque hang-ups on interviewers, I found it hard to believe that men and women in Arab countries were any more negatively disposed toward opinion surveys than people anywhere else.
Surveys about Surveys
In 2017, these ideas crystallized into a successful grant proposal for a study of public attitudes toward survey research in Qatar—a survey on surveys—with co-investigators Russell Lucas (Michigan State University), Mark Tessler (University of Michigan), and Jonathan Forney (Forcier Consulting). Improbably, perhaps, the use of questionnaires to assess views toward opinion polls, commonly referred to as survey attitudes, dates to the 1950s[ii] and has produced a sizable literature. But, prior to our study, existing research on survey attitudes had been advanced solely on the basis of data collected in Western Europe and, despite many decades of investigation, still had produced no theoretical consensus regarding the number and character of the specific attitudinal dimensions that comprise views toward surveys. Nonetheless, results of previous studies broadly agreed in showing that more positive attitudes toward surveys are associated with more cooperative respondent behavior and thus to better survey data.
Extension of this work to Qatar was significant not only in gauging survey attitudes in a very different context—one representing exactly those social and political factors said to undermine the quality of Arab opinion data—but also because Qatar’s extreme demographic diversity allows comparison of survey attitudes across the various cultural-geographical groupings represented in the Qatar population. Our survey sample contains respondents from more than 50 countries residing in Qatar, including more than three-quarters of Arab League states. This diversity allows a direct test of the idea that Arabs as a broad category possess more negative perceptions of surveys than people from other national or regional backgrounds. The data also offer insight into the origins of potential negativity toward survey research. To the extent that Arabs do hold more skeptical views of surveys, is that because surveys are viewed as irrelevant to policymaking, because they are seen as invasions of privacy, because their results are considered unreliable, because they are too cognitively burdensome, or for some other reason that the survey attitudes literature has identified?
As it turns out, these were the wrong questions to be asking. Or, rather, they erred in their premise. Results from our study of survey attitudes in Qatar show not only that Arabs tend to hold positive attitudes toward surveys on all measured dimensions, but that their attitudes are more positive than those held by individuals from other cultural-geographical regions represented in Qatar, including South Asia, Southeast Asia, and—ironically—the West. The findings also revealed a second surprise: a new dimension of survey attitudes not observed in previous studies conducted in Europe. This dimension captures perceptions of the potential positive or negative purposes of surveys: the use of surveys for purposes of political manipulation or surveillance, rather than political voice and participation. On this aspect of surveys as well, the data show that Arabs in Qatar tend to hold more positive views on average than non-Arabs.
However, two experiments embedded in our survey demonstrate an important divergence between Arabs and non-Arabs in the effects of survey attitudes on their actual response and nonresponse behavior in surveys. A conjoint experiment presented respondents with a hypothetical survey of randomized mode, length, topic, and sponsor and asked them to rate their likelihood of agreeing to participate. Results show that, while both Arabs and non-Arabs are sensitive to the mode and length of a survey, only Arabs are influenced in their decision to participate by the survey sponsor, being less likely to participate when a poll is conducted by an international organization or private company. (Notably, state sponsorship did not depress participation.) In addition, Arabs are influenced in their decision by latent attitudes toward surveys generally, irrespective of the parameters of a particular survey request. Regardless of the specific characteristics of a survey, Arab respondents are less likely to participate when they possess more negative views about enjoyment of surveys, the reliability of surveys, the time and cognitive costs of surveys, and the political purposes of surveys. The participation decision among non-Arabs, by contrast, is impacted only by their overall views about the enjoyableness of surveys.
An analogous result obtains in the case of the second experiment embedded in our study, which examines how survey attitudes influence respondents’ willingness to complete the full interview schedule when presented with an easy opportunity to cut short the survey. We call this the “birthday experiment.” Respondents are informed that, due to the length of the final section of the survey, only one-half of respondents will be asked to continue: those whose birthdays fell in the previous six months. We expected that some individuals who wanted to terminate the interview would lie about the timing of their birthday, and that respondents with negative views about the cognitive and time costs of surveys might be especially likely to drop-out. In fact, this turned out to be true only for non-Arab respondents. Among Qatari and non-Qatari Arabs, the likelihood of early termination was determined instead once again by overall views of the reliability and purposes of surveys, rather than negative impressions about the time and effort it takes to complete surveys.
Our study of survey attitudes in Qatar thus gives reason for optimism while also identifying some challenges to ensuring high-quality opinion data from the Arab region. On the one hand, it helps to dispel persistent worries that authoritarian institutions, traditional social norms, and public unfamiliarity with surveys instill negative views of the entire survey research enterprise among ordinary Arabs. At the same time, however, our findings show that the Arab context gives rise to specific concerns about surveys that may not exist elsewhere, namely the use or manipulation of survey research for political purposes. Equally significant, it shows that Arabs’ response and nonresponse behavior in surveys is disproportionately sensitive to subjective impressions about the general reliability and intentions of surveys—concerns that are not seen to impact the survey-taking behavior of non-Arabs in Qatar. Arabs on average are more positively inclined toward surveys than other groups, yes, but their participation in survey research is also mediated by a different and wider range of generalized survey attitudes.
Some Practical Implications for Researchers
This article has highlighted recent collaborative work exploring how people in the Middle East and North Africa view survey research, and the ways in which their attitudes differ from those of other cultural-geographical groups around the world. Equally important, this research is interested in understanding the practical impact of these orientations on actual survey behavior. To date, these topics have not been widely studied in the MENA context, although they are motivated by the same basic concerns surrounding the region’s survey-taking climate that have prompted other methodological work into sponsorship effects, third-party effects, and interviewer effects in Arab opinion data.
Our substantive findings serve to dispel certain stereotypes about the unique aversion and even hostility of Arab men and women toward survey research, whether in general or compared to Western or other populations. Yet our results do also give reason for further reflection and empirical investigation about the particular ways that survey research is experienced, perceived, and understood in MENA countries, and what, if anything, these context-driven concerns and conceptualizations imply about the nature and quality of data reported in public opinion surveys. The foregoing discussion suggests some practical lessons for survey researchers working in the Arab region.
Reorienting survey design concerns. Survey practitioners everywhere spend significant time deliberating basic survey design questions, including the mode and target length of an interview. A common concern, for instance, is that an interview schedule is “too long” and may increase respondent fatigue and therefore survey error via nonresponse, satisficing (providing quick answers without consideration or comprehension of the question), or early termination of the interview. Often many a tear is shed in cutting questions to shorten the instrument below an acceptable, though usually arbitrary, limit. Survey researchers working in Arab countries also frequently debate whether a certain topic is better suited for telephone (or Internet) versus face-to-face administration, based on the assumption that some topics may be difficult to probe in an in-person interview because they place too high a burden on respondent privacy.
“More generally, our experimental findings suggest that MENA survey researchers should be less concerned about survey design features that may increase perceived survey burden—whether from a cognitive, time, or privacy standpoint—and focus instead on aspects that may instill negative impressions about a survey’s reliability or purpose.“
But the results of our study show that these two basic survey attributes—length and mode—are less influential in shaping Arab (and indeed non-Arab) survey-taking behavior than the survey’s topic and sponsor. More generally, our experimental findings suggest that MENA survey researchers should be less concerned about survey design features that may increase perceived survey burden—whether from a cognitive, time, or privacy standpoint—and focus instead on aspects that may instill negative impressions about a survey’s reliability or purpose.
Signaling survey reliability and intentions. Impressions about the reliability or purpose of a survey may be formed from overt attributes, such as the stated sponsor of a survey, and some experimental work has sought to understand the impact of sponsorship on nonresponse.[iii] Yet positive or negative views about a survey’s integrity and purpose may also arise from more subtle cues accumulated over the course of an interview, including from the topics included in a questionnaire, the ordering of questions, response options, the names of survey sections, and myriad others. For example, a survey module on international affairs that asks only about attitudes toward the United States is likely to signal, correctly or incorrectly, American involvement or interest in the survey. A battery measuring religious values and practice embedded in a section on gender stereotypes is likely to signal a researcher’s belief that Islam encourages negative views of women.
Such concerns about question wording and ordering are typically conceived as pertaining to the socio-political “sensitivity” of a questionnaire—reluctance to respond or give accurate answers due to social desirability pressures or doubts about anonymity in an authoritarian survey environment. However, the effects of negative views of survey reliability and purpose operate differently and more fundamentally, by raising questions about the motivations of the very survey itself. They are also potentially more insidious than sponsorship effects, because negative judgments about the integrity or reliability of a survey may form only once an interview has begun, introducing survey error not only through systematic nonresponse but also response bias and motivated underreporting.
Encouraging public confidence in surveys. Another way that negative views of survey reliability and intentions may form in MENA populations is through exposure to unscientific polls and/or surveys designed to support a particular political agenda.[iv] Such unscientific or politicized surveys are unfortunately a common occurrence in the Arab world, enabled by a lack of institutional capacity and monetary incentives to carry out high-quality surveys. As a result, survey research is too often exploited to promote the domestic and foreign policy aims of MENA governments and outside political actors, with the aim of creating the false impression of public support for or opposition to an actor or policy position.
Our finding that Arabs in Qatar are especially reluctant to participate in surveys conducted by commercial polling firms and international organizations, compared with a university or government institution, suggests that these types of sponsors are associated with negative survey purposes. In this way, hesitation to participate in surveys in the MENA region may stem from generalized worry that survey results will be manipulated or used for nefarious purposes, rather than a survey’s sponsor or topic itself. While much work remains to be done to understand the root causes of attitudes toward surveys in the Arab world, it is hoped that this research can set the agenda for the next wave of studies.
[i] J. Gengler, M. Tessler, R. Lucas, and J. Forney. (2019). ‘Why Do You Ask?’ The Nature and Impacts of Attitudes toward Public Opinion Surveys in the Arab World. British Journal of Political Science. The study was made possible by a grant (NPRP 9-015-5-002) from the Qatar National Research Fund, a member of The Qatar Foundation. The statements herein are solely the responsibility of the authors.
[ii] G. Sjoberg. (1955). A Questionnaire on Questionnaires. Public Opinion Quarterly 18, 423-427.
[iii] E.g., D. Corstange. (2014). Foreign-sponsorship effects in developing-world surveys: Evidence from a field experiment in Lebanon. Public Opinion Quarterly 78, 474–484.
[iv] J. Gengler (2017). The dangers of unscientific surveys in the Arab world. Washington Post. October 27.