Research on Research: Mobile Versus Online Results
Posted May 21, 2013
The purpose of this research-on-research study is to document how, and to what extent, respondents completing research on Smartphones differ on their attitudinal ratings and behavioral data reported from those found among the general online population.
There are a number of reasons why mobile technology is causing everyone to take notice, and one should not be ashamed for jumping on the mobile band wagon. For starters, traditional advertising (TV, radio and print) has been on a steady decline which means viewers and listeners are stepping away from these mediums to adopt new digital service platforms like Pandora. The result is billions of advertising dollars being shifted away from traditional channels to mobile. According to a recent report from eMarketer, U.S. mobile ad spending is expected to reach $7.19 billion next year and nearly $21 billion by 2016 - a significant upward revision based on previous forecasts. This rise in Ad spending is even causing some analysts to predict that local mobile advertising will surpass what is being spent by national advertisers.
Ad spending is not the only area where mobile is causing a stir - mobile technology is being used in a number of health-related initiatives, especially in developing countries. The technology has been used to diagnose patients, transfer remittances to family members, promote transparency in elections, maintain refugee databases in disasters, provide pricing information to farmers, and many other uses. In some developing countries, the technology is widespread - often used by at least 50% of the population. For example, in Kenya, more than half (29 out of 42 million) of the population are mobile users.
There's no doubt mobile technology is here to stay and will continue to make a significant impact and revolutionize how we, as a society, communicate and conduct business; and the Marketing Research industry is no exception. Mobile research has been dubbed by many as "The Next Big Thing" for data collection. While there are many different ways to use mobile phones to collect data, the more broadly mainstream approaches (e.g., IVR and texting) tend to be mundane and cumbersome. On the other hand, the more fun and engaging methodologies (e.g., Apps or mobile Web) require newer phones and data plans, and data collected via Smartphones (about 60% of mobile phones currently in use in the U.S.) is not deemed as "representative" of the population as a whole.
The purpose of this research-on-research study is to document how, and to what extent, respondents completing research on Smartphones differ on their attitudinal ratings and behavioral data reported from those found among the general online population. In addition, because everyone does not have a Smartphone, but Internet access is nearly ubiquitous, it is important to understand the demographic composition of both of these samples. Finally, a recommendation on the usage and appropriateness of mobile research data collected via Smartphones is made.
To achieve a fair comparison, the same survey was programmed both for online administration as well as for the Smartphone App, SurveySwipe™. The survey included only 10 questions to accommodate the mobile platform. For both surveys, 3500 panelists were invited to participate in the research study. Incentives were nominal and of comparable monetary value.
Panelists who downloaded the SurveySwipe Application were alerted through real time push notification or simply by launching the app to see if a new survey was available. In total, 619 panelists started the mobile survey with 591 completing it (a 95% completion rate). Most responses came in during the first few days of availability.
Members of the qSample online panel received email invitations to complete an online survey. In total, 309 respondents finished the survey out of the 318 panelists who started it (a 97% completion rate). The study was left open a full week to achieve a minimum of 300 responses.
No statistically significant differences between the two modes of data collection were found in terms of stated usual brand of gum chewed. For the most part, the similarity of results among the mobile and online samples continue to hold when viewing usual gum flavor.
A majority of the respondents tend to chew the same brand/flavor until tired of it. A few never change brand or flavor.
The general pattern of responses for both samples is very similar. Online respondents are slightly more likely than mobile respondents to stick with a brand/flavor longer before switching. For the mobile respondents, 19%are likely to chew one brand/flavor for one month or less, compared to 13% for online respondents. 18% of mobile respondents stay with one brand/flavor from two months to two years compared to 29% for online respondents.
Online respondents chew slightly more gum: 2.8 packs per month on average vs. only 2.1 for mobile respondents. 23% of mobile respondents chewed more than one or more pack per week, compared to 36% for online respondents.
More online than mobile respondents are likely to buy their gum from the candy aisle of the store.
Two-thirds or more of both types of respondents tend to buy gum as individual packs.
In terms of likelihood of trying a new brand, the distribution of answers are very Similar between both groups with only 20% being unlikely to change brands. Online respondents are more inclined to say they are Extremely Likely to try a new flavor (29%). In contrast, mobile respondents say Somewhat Likely more often (36%). As far as trying new flavors from a previous brand, the pattern of the results is again very similar among respondents of both modes of data collection.
In general, respondents completing the mobile surveys are younger, are more likely to be employed full-time and are more likely to be single. In contrast, respondents completing online were more likely to be retired and married.
Eight percent of the mobile respondents were under 18 versus none for online respondents. Only 1% of mobile respondents were 65 or older versus 8% for online. The average age for mobile respondents was 31.6 years as compared to 41.5 years for online respondents.
Males were somewhat more likely to respond to the mobile survey while females were somewhat more likely to respond to the online survey.
While the results for Whites, Asians and others did not differ significantly on mobile versus online responding, Blacks were more likely to respond to the online survey and Hispanics were more likely to respond to the mobile survey.
Respondents who responded to the mobile survey were more likely to be employed full-time or to be students. Respondents who responded to the online survey were more likely to be employed part-time, homemakers, or retired. Mobile respondents were more likely to be single and online respondents were more likely to be married.
Summary of Key Findings
The pattern of results for both samples was very similar for both brands and flavors chewed. In addition, a majority of both samples tend to chew the same brand/flavor until they get tired of it and then switch. Online respondents, however, chew more frequently (36% chew a pack a week or more versus 23% for mobile respondents) and are more likely to stick with a brand/flavor longer. With regard to purchasing, the two groups are highly similar; most purchases are individual packs found in checkout lanes.
The two samples are equally likely to try a new brand, but online respondents are slightly more likely to say they will try a new flavor. Both groups are equally likely to say they would switch back to a brand they used to chew if that brand introduced an interesting new flavor.
The mobile sample was younger, had higher incomes, and were more likely to be working full-time and be single. In contrast, the online sample had a higher proportion who were retired and married.
Conclusions and Recommendations
Few differences were found between the mobile and online results in terms of attitudinal and behavioral measures. The pattern of results among both samples was very similar. Only one of the statistically different results was related to an attitudinal rating. A few small behavioral differences were found (i.e., in the amount of gum chewed and the likelihood to buy gum from the regular candy aisle). Finally, the one attitudinal rating difference is only 7 percentage points, statistically significant due to the sample sizes in this study but probably not actionably different. Some differences may be noted purely because of the probabilistic nature of statistical testing. These highly similar results were found despite significant and consistent differences in the two samples' demographics.
Mobile data collection has an advantage over on-line surveys in cooperation rates (i.e., likelihood to participate) and in speed of response. Twice the data was collected in half the field time. The mobile survey had to be shut down earlier than the online one, which was left open nearly a week to achieve the desired sample size of 300 completed surveys. For short surveys, mobile data collection can reasonably replace online data collection. In addition, the quality of data could be expected to be superior because the methodology is still a novelty and respondents seem more engaged. Because of this, field times can be shortened as well.
qSample combines years of professional experience with the latest achievements in research and sampling technology. We specialize in online and offline data collection, using a suite of innovative custom software solutions in the form of web, mobile, iPad and idea generation technology to collect valuable insight from our respondents.
qSample specializes in developing and managing specialty research panels that cover a wide range of audiences - Gamers, Mobile, Building contractors, Homeowners, Likely Voters, College students, Boomers, AppDevelopers, Frequent Travelers, Small Business Owners, and Wine Enthusiasts, just to name a few. These specialty panels are developed, recruited and managed in-house, giving our clients access to more than 5 million respondents in over 42 countries who are pre-screened and double-opted in for participation in a wide range of research studies.
About MAi Research
Over the past 30 years, MAi Research has developed and continues to refine a highly disciplined program management process, one that delivers flawless data and impactful customer insights efficiently, quickly and with high client value. Our methodologies are designed to give our clients full strategic control over the research at each step without over-burdening them with executional considerations and implementation details.
Our highly skilled marketing research professionals are, first and foremost, business experts who provide insightful, clear answers to your most complex and often confounding questions. Our proprietary research approaches yield provocative insights and create effective pathways to your next level of in-market success. MAi Research's passionate commitment to clients, their customers, and to innovative research excellence assures that their clients are moving in the right direction.