Has Gongos Research Proven The Validty of Smartphone Research Or Not?

UPDATE 06/29: Due to the overwhelming interest in this topic we’re working on putting together a virtual debate featuring Dr. Alioto from Gongos Research, Ray Poynter of The Future Place, Reg Baker of Market Strategies (who posted on this topic last week) and myself on July 7th at 12:00 PM EST. Read all about it & register to attend here. See you there!

Yesterday Ray Poynter posted a response to the press release by Gongos Research regarding the results of a new study they conducted on the validity of smartphone research. You can find the original press release here.

Ray questioned the claim in his post in his usual highly erudite way:

OK, somebody had to blink! For the last few days there have been several tweets and comments using the phrase “Smartphone surveys prove their validity in marketing research”. When I saw the first post I immediately classed it as harmless hyperbole. But when I saw that MSNBC had used the same headline (14 June 2011) I felt I had to shout out “The Emperor has no clothes”.

The stories seem to track back to a press release by Gongos Research, released on 14 June 2011. The press release includes the phrase “a new study proves that smartphone-based survey data is statistically comparable to online survey data.”

This claim in linguistically and methodologically ludicrous, a single study cannot prove that something happens, it can show that it sometimes happens, but that does not prove a positive. A single, well designed, study can show that something is NOT true. For example, the claim that all swans are white is disproved by finding one black swan, but finding one more white swan does not prove all swans are white.

If we assume, for the moment, that the press release means that Gongos Research conducted a study, looking at a wide range of questions types, in one or more markets, with one or more types of customers, and that the study showed the results were acceptably similar, then we would not have proof of the validity of smartphone research. We would have evidence that smartphone surveys can sometimes work. If more tests are conducted (and ideally other agencies should be willing to pay for them) we can start to find out if smartphone surveys are ‘often’ (or better still ‘usually’) acceptably similar to other modalities. If we had the results of say 20 studies and they all showed that smartphone research was acceptably similar to other modalities we would not have proved it, but we would feel reasonably confident about using it, even 10 out of 10 positive results would make us pretty happy about trying it for a live study.

I do hope that Gongos Research publish their research as it is potentially really helpful to the industry, as evidence about the viability of smartphone research.

Dr. Michael Alioto, the principal author of the report sent me his response and posted it on Ray’s blog. Here it is:

I appreciate your comments, Ray. Indeed, our intent was to shed light on the currently existing body of evidence in our industry – not claim a “proof” in the Philosophy of Science or mathematical sense.

Much of the literature surrounding the use of mobile or Smartphone devices for survey research is mixed. This study indicated that data can be collected and compiled in multi-mode formats with confidence, which enabled us to support some of the accepted approaches and question others.

But let’s not lose sight of the real issue. While we have solid evidence to support the use of multi-modal platforms for research, the question of migration from online to smartphone as a preferred platform is still to be determined.  Many of us in the marketing research discipline agree that online surveying has its issues (biases associated with representation, scale uses, artificial interviewing environment, etc.).  These issues have become increasingly apparent as the online platform has been less able to solve “puzzles” in the Kuhnian sense of the term.  The less puzzles solved, the more we are inclined to view the smartphone platform as a potential replacement for online surveying (assuming that the smartphone platform will be able to solve more of the puzzles identified by the online methodology and other potential platforms, such as social media, etc. are not direct competitors for the next platform paradigm).

The importance of this study for the marketing research industry is that we were able to rigorously test a number of assumptions and items between the two separate platforms.  But this is not the end.  Similar to your thoughts, we welcome other research agencies to join in rigorously testing data and research elements between the two platforms.  Based on our findings, we believe that these additional tests will support the use of smartphones as a valid survey platform.

The critical nature of the future of marketing research surveying is at hand. With technology progressing forward and response rates continuously falling, we need to be able to gather valid and reliable information from our respondents. We stand behind smartphones being a valid avenue to accomplish this goal.

We plan on rolling out the key findings from this study at the MRMW11 conference. Ray, we would love to have you join us and contribute to the debate and good discussion.

So, there is the current state of what is happening leading up to a much needed and critical debate. I’d like to suggest to both parties that perhaps we continue this at the Market Research in the Mobile World Conference next month, or if that is not possible perhaps stage a virtual debate around the same time?  It’s time to tackle these issues head on; the future of the market research industry is too dependent on mobile and social media-based technologies  for us not to exert the needed time and energy to answering these questions and moving on.