Secondary data answers marriage equality question

The ‘polling’ is wholly different from the process adopted by Australia's federal parliament in 2004, when without consulting the electorate, it amended the Marriage Act to expressly ban same-sex marriage, including refusing to recognise those lawfully solemnised in other Commonwealth countries.

A dubious research purpose

The upshot is that $120m+ is being spent just to decide if the Parliament will deal with marriage equality, but not to determine its outcome. In a commercial comparison, it would be like a large company spending $1m on research (alone) just to decide whether it’s going to consider whether to introduce a new product, and not to decide whether to actually introduce one or not.

To add insult to injury, the $120m+ expenditure was quite unnecessary, because we know the answer to the ‘poll’ question anyhow. We know because there is already ample, robust secondary data.

Primary versus secondary research

When I ask my market research students which should be done first, primary or secondary research, most say primary. It's almost a trick question because it sounds right in an ordinal sense. But it’s wrong. Primary research is a study conducted primarily for your exact research needs. It can cost a lot, and takes time.

Secondary research is the sourcing and analysis of existing data — which may not have been collected directly for your primary purpose, but nonetheless may be informative and helpful. It usually costs a lot less than primary research and can even be free. It’s also relatively quick.

Good secondary data can sometimes answer your research question quickly and cheaply.

So you usually conduct secondary research first. You may find helpful clues to answering your research question, or at least helping to frame how to conduct your primary research. Sometimes it might answer the question directly and immediately without the need to conduct primary research.

Secondary data on marriage equality opinion

And that’s exactly the case for marriage equality law reform in Australia. Setting aside unscientific readership engagement polls run by media outlets, there is a wealth of robust and professional data on national voter attitudes toward marriage equality (Figure 1).

Figure 1: Australian voter support for marriage equality
 

Immediately apparent from the chart is that since 2010, every single poll has returned a result above 50%. Also immediately apparent is that support is increasing substantially over time.

Quality of research data

Some remarks about data quality are pertinent, given the range of results (e.g. by twenty percentage points from 56% to 76% in 2016). Heads-up: I’ve circled data points with inappropriate methodology, usually in relation to the question asked. It’s important to ask a reasonably-worded question, balanced in both the positive and negative, directly about changing the law to permit marriage of non-heterosexual couples.

The HILDA study (black line) is run through the University of Melbourne. It has impeccable methodology, with rigorous sampling, large sample size, and a battery of same-sex questions worded in either the positive or negative. It shows a substantial and sustained increase in support for marriage equality over a decade.

An early hiccough in AES questioning

The AES study (dark blue dotted line) is run through Australian National University. Its 2013 result of just 51% is heavily biased by the question that was asked:

“Same-sex marriages should be prohibited by law.”

Some common forms of question construction can harbour egregious bias.

This question suffers from substantial biases: bias from wording as a statement rather than a question, promoting normative bias (reiterating only the current familiar situation); from ambiguity bias (who should be permitted to prohibit – e.g. churches?); and from acquiescence bias (stated as agreement rather than asking 'agreement or disagreement').

The 2016 AES study, in contrast, asked a much more neutral and appropriately future-focused question:

“Do you personally favour or oppose same sex couples being given the same rights to marry as couples consisting of a man and a woman?”

This more appropriate form of questioning in 2016 returned 71% in favour, extremely close to the trend in the HILDA surveys.

Early hiccough in Fairfax/Ipsos questioning

In 2013, a Fairfax/Ipsos poll (orange line) asked Australians:

“When you think about the rights of same-sex couples, which of the following comes closest to your personal opinion: (a) Same-sex couples should be allowed to marry legally, (b) Same-sex couples should be allowed to obtain some kind of legal recognition, but not to marry.”

The question is biased by failing to ask in a way equivalent to heterosexual couples: what is the respondent’s opinion in regard to opposite-sex couples obtaining legal recognition without marriage? The two statements are given as a dichotomy, rather than asking about equality in principle: rather asking about equality versus an unequal and currently lawful substitution. It returned 54% agreement for marriage equality.

In 2015 and 2016, Fairfax/Ipsos asked a much more balanced question:

“Do you support or oppose legalising marriage between same-sex couples?”

Those polls returned 68% and 70% respectively.

Heavily biased VoteCompass question

Many commentators have held up VoteCompass polling (red line) as robust, because it has a sample size in the many hundreds of thousands. But, in attitude research as in so many other facets of life, size isn’t everything. VoteCompass asked, in 2013 and 2016, this question:

“Marriage should only be between a man and a woman.”

This question suffers from a number of major biases. Firstly, it’s a normative statement (not question) of the present and past rather than a question about future law reform.

Secondly, it’s worded in the presumptive negative, leading to substantial acquiescence bias.

Thirdly, it asks a simple ‘moralistic’ question rather than one about changing law. Some respondents may think that marriage ‘should’ be between a man and a woman, but still agree that same-sex couples could be afforded the same right (i.e. a preference rather than a prohibition).

Massive samples sizes like those of the VoteCompass polls do nothing to 'correct' for egregious question bias.

And fourthly, it leaves the “alternatives” field open far wider than “same-sex marriage”, because in referencing only “a man and a woman”, it fails to articulate specific alternatives. Opponents of marriage equality have been surmising that if the law were changed, three or more people could marry, people could marry pets and companies, and, according to at least one conservative politician, people could even marry bridges.

Clearly these alternatives are unwarranted worries, but the biases led to the VoteCompass poll returning just 52% in 2013, and by far the lowest marriage equality approval figure of 56% in 2016.

And to make it clear in case there's any doubt, the massive samples sizes of the VoteCompass polls — in the hundreds of thousands — do nothing to correct the egregious bias in the question itself.

Morgan sampling bias

Morgan research has returned two of the highest results, including the highest (76%) in 2016. This is most likely caused by their sampling method: phone text (SMS) messaging. More younger than older people are likely to respond to SMS messaging, and younger people are far more supportive of marriage equality. It may also be that more socially-oriented people respond to SMS; people who are more likely to be accepting of others as they are.

While it’s normal practice to adjust the raw sampling demographics to match census demographic rates (e.g. matching age and gender), there’s only so much that post-fieldwork sample weighting can do.

Essential sampling bias

Essential research has, since 2012, consistently returned lower than average results. They recruit respondents from a fixed proprietary database, which they recruit with “offline methods.” The recruitment methods may attract a larger proportion of social conservatives, even though demographically the sample seems balanced. (Demographic ‘balance’ does not equal psychographic balance, especially on socio-moral issues.)

Indications are that attitudes amongst the Essential database pool are changing (trending upwards), but more slowly than the overall population’s.

A very close estimation

Taking the most reliable polling result together — HILDA, AES 2016, and Fairfax/Ipsos 2015/2016 — and extrapolating the trend out one year to 2017, the answer to the question of public support is this:

“More than two thirds of Australians support marriage equality in 2017: at 99% confidence, support is 71% ±2%.”

No opposed electorate

Given the massive sample size of the VoteCompass polls, scholars have been able to calculate whether, in 2016, any federal electorate (Division) in the nation has a majority opinion against marriage equality. University of Melbourne’s Andrea Carson and colleagues found just one electorate, Maranoa, with a bare majority (53%) opposed (see Figure 2 below).

However, we’ve already established that the VoteCompass question (statement) was heavily biased and that it returned by far the lowest support for marriage equality in 2016 (56%), 8% lower than the next-lowest result (Essential at 64%) and 13% lower than the average of the robust HILDA, AES and Fairfax/Ipsos polls (69%).

On that basis, we can confidently conclude that:

“Public opposition to marriage equality is in the minority in every federal Division across the nation.”

Conclusion

Analysis of professional and scholarly polls over more than a decade show that Australian support for marriage equality has been rising strongly. In 2017, we can be 99% confident that support is 71% ±2%, and opposition is in the minority in every electoral Division across the nation.

Had the Australian government consulted freely-available, robust secondary data, there would have been no need to squander $120m on primary research.

Had the Federal Government bothered to check the robust secondary data available for free, it would have found it unnecessary for any reason other than political, to conduct a ‘poll’ costing more than $120m.

I say ‘poll’ in inverted commas (and certainly not a 'plebiscite'), because the ‘thingy’ being run by the Australian Bureau of Statistics, at the government's direction, is a methodological farce — failing to follow some of the most fundamental standards of social and market research. But that’s a subject for a separate discussion.

--------

Figure 2: Carson et al VoteCompass electoral Division attutides toward marriage equality
Note: Labor-held seats in red, Coalition-held seats in blue
 

Carson, A, Ratcliff, S & Durfresne, Y 2016, Same-sex marriage debate in Australia: Public opinion and policy congruence, Australian Political Studies Association, pp. 36.

 

Keywords: 

Australia

|

Analysis

|

Bias

|

Market research

|

Poll / survey

|

Questionnaire design

|

Secondary research

• Log in or register to post comments