ICYMI (In Case You Missed It), the following work was presented at the 2015 Regional Conference of the World Association of Public Opinion Research (WAPOR) in Doha, Qatar. Fatimah Al-Khaldi, a researcher on the Policy Unit team at Qatar University’s Social and Economic Survey Research Institute (SESRI), presented “Survey Experiment on Attitudes & Perceptions of Women in the Political Sphere” as part of the session “Questionnaire Design” on Sunday, March 8th, 2015.
Post developed by Fatimah Al-Khaldi.
To what extent are survey results relating to gender attitudes accurate and reliable? Often survey questions are worded in ways that are more positive for a man than a woman. For instance, consider the following items which appear in questionnaires from the World Values Survey:
“when jobs are scarce, men should have more right to a job than women
“on the whole, men make better political leaders than women do”
In the questionnaires, statements that are in contradiction with gender stereotypes are almost entirely absent.
To investigate the impact of this question wording, I conducted a survey experiment using Amazon’s Mechanical Turk. Survey respondents, comprised of a sample of adults living in the United States, were divided into two samples. Respondents in each sample were asked to what extent they agreed or disagreed with a set of statements.
Sample 1 received statements worded to emphasize gender stereotypes:
With the exception of the fourth statement, the majority of Sample 1 respondents strongly disagreed with the statements. In all statements, women were more likely than men to strongly disagree, and respondents who identified themselves as moderate or liberal respondents were more likely than self-identified conservative or very conservative respondents to strongly disagree.
Sample 2 received statements worded in contradiction of accepted stereotypes:
For each of the five statements, approximately half (or nearly so) of Sample 2 respondents answered neutrally (“indifferent”), with women being more likely than men to express neutrality. Men were more likely than women to strongly disagree with the first statement. Respondents who identified themselves as very conservative were more likely to strongly disagree on the first and fifth statements.
The results of the experiment show a clear difference in response distributions across the two samples, confirming that the wording for questions on gender attitudes does indeed matter.
Methodological Statement. The survey experiment was conducted in English on the Internet via Amazon’s Mechanical Turk from March 8-16, 2014, using a sample of 328 adults who were 18 years of age or older and living in the United States (citizens and residents). The work was part of a class project at Johns Hopkins University.
(Edit, applied March 9, 2015: Tables were previously in incorrect order.)