Online Dating: Usability Study Report
This study evaluated two top online dating sites for usability - eHarmony and Match.com. Users shared their experiences and perceptions of the two sites in interviews and eye-tracking research was conducted. The results were interesting and pointed out issues with technology facilitating emotional connection and chemistry between users.
Statement of Limitations
This report summarizes conversations held with a total of 16 people (8 eHarmony and 8 Match.com users). The comments, quotes, and opinions in this document reflect only the views of these users and not those of eHarmony or Match.com who were not themselves involved with this research in any way.
The intention of this report is to highlight broad themes and provide insights as to how these sites are used by the users we spoke to. The observations in this report reflect the views of these users which, while accurate for this population, may not be representative of the overall population.
To protect the privacy of the users whose profiles have been used in this report, we substituted the main pictures with those from a photo library in most cases, and in other cases we blurred the pictures. All names and locations have been blurred.
Objectives and Methodology
What was this project about?
With Valentine’s day approaching we wanted to see what users thought of two of the most popular and heavily promoted online dating services: Match.com and eHarmony.
Ultimately we were interested in answering the following questions: How effective are computers at helping users find love? Given that the sites use different approaches to online dating, was one service considered superior to the other? If so, why? How do users identify potential matches? How do users assess the relationship potential of their matches?
What did we do?
We recruited 16 users in four groups: 4 men and 4 women who use each site. All users were between 30 and 40 years old All users had to be active (i.e. they were paying members of either site), were currently seeking dates, and had been on at least one date via their site in the last month. Users were interviewed at office in New York City or over the internet using WebEx and telephone We asked all users who were interviewed at Catalyst to eye-track a Match.com profile we created. Men were shown a female profile and vice-versa. We then explored the users general experiences of online dating and their specific experiences with the site they use. Next, users were asked to log into their account and demonstrate what they would typically do to find matches, assess matches, and then communicate with matches. We were interested in their overall views of the dating experience on their respective sites. Where users had used both sites, we were interested in their comparative observations.
In general, users told us that online dating sites, including eHarmony and Match.com, come up short on their claimed ability to make good romantic matches.
Specifically, there was little confidence in the idea of using technology to replace the very complex process of developing human relationships. The simple forms and crude matching techniques on these sites led many users to conclude that online dating sites are best viewed as a pool of potential matches which include some rough screening tools.
Despite these shortcomings people use the tools mainly because the traditional approaches – bars, clubs, family, friends, via work, etc. aren’t working for them.
eHarmony was preferred by people who would prefer a high degree of handholding – it’s suited to beginners and people with lower self confidence in initiating communication. Experienced online daters use eHarmony because “it’s another pool of potential dates,” however, they thought the profiles were “formulaic” and “hide people’s individuality.” Also, the length of the guided communication process can be painful in that it can literally take weeks before you are able to communicate openly with a potential match.
Match.com was thought to offer a good compromise between the restrictions of eHarmony and the “wild west” of certain free sites such as OKCupid. However, most users felt that the matching algorithms here were “rough” and could be made more effective. As a result, users often stated they felt they had to work harder than necessary because of the large number of poor matches returned.
eHarmony and Match.com each provide very different ways of identifying matches with Match.com providing a much larger number of options. For the most part Match.com users said they found the basic search useful but many showed us that they often add several “deal-breaker” criteria (such as age, wants kids, etc.) to the advanced search in the “more search options” interface.
Other than the main search, Match.com users had a variety of techniques they used to identify potential matches. Of these, the Daily5 (a computer generated match based on answers gathered from quick poll results) was the next most commonly used method of identifying potential matches. Although it wasn’t thought to be particularly effective it was felt to be a fun approach. A variety of other contact methods were mentioned such as “Who’s viewed me?” (provides a list of people who clicked on a member’s profile), “Winks” and email contacts but usage of these was mixed amongst our users.
We also probed on some of the readymade computer matching methods Match.com offers, notably Mutual Matches (which matches users who have each described each other as the person they are looking for) and Reverse Matches (which identifies Match users that have described the searcher as the type of person they are looking for). The Mutual Match search had been used by most users but only a few used it with any frequency citing that they felt their personal matching techniques were more effective. Few people understood the Reverse match which reflected its low usage level.
On eHarmony locating potential matches is limited to only those people that the site determines to be a match. A few users saw this as an advantage but many users felt this was a significant weakness of the site – specifically as eHarmony does not provide any feedback as to the degree of compatibility or where the compatibility is. Given this lack of transparency, many users felt the site would be improved if they could also define their own searches.
Initially, we were quite surprised to observe that men and women assess the information in match profiles almost identically regardless of the site they are on. But as we observed over the course of our study, online dating appears to be governed primarily by “facts” at the start of the process which then quickly give way to more complex factors.
Assessment is typically performed as follows: (1) Look at pictures; (2) Look at basic information for high-level “deal-breakers” such as age, distance from each other, and kids; (3) then, finally, consider user created descriptions for “human touch points.” While the last step is where likely compatibility is most strongly identified, it was very apparent that most users felt they were wasting time if potential matches don’t pass muster in the first two stages. A typical comment was “Why would I want to learn if a person is interesting if I don’t find myself physically attracted to them or that they are not interested in having kids if I am?”
Initial assessment is generally quick, ranging from a few seconds if pictures do not suggest physical attraction or a “deal-breaker” is encountered, to about 45 seconds (for both men and women) for a more complete profile.
Eye-tracking confirmed what users told us about the Match.com “computer-based chemistry assessment” tool - users paid it very little attention, if any. eHarmony users were also very skeptical about the effectiveness of the “29 levels of compatibility calculation.”
Many users talked about their confidence level improving over time in regards to their ability to identify better matches for themselves from information provided in user profiles. Several users recounted almost identical stories of their early online dating experiences where they failed to observe “red-flags” in a potential match’s profile as they were excited about the prospect of going on a date. After several dates it seems clear that many users develop more acute assessment abilities. As one user told us “I want to go on dates with people I think I have potential with and not waste my time with people I should have avoided..."
Communicating with Matches
This is where the other main difference between eHarmony and Match.com exists and is a significant factor for some users as to whether they use one site or the other.
On Match.com, users are able to begin communicating immediately they locate a potential match. For many, this is the preferred approach. The matched pair remain “hidden” from each other until they choose to “reveal” themselves to the other party. This typically occurs after several email exchanges as they prepare to meet for a date. If an offer to communicate is not reciprocated then the parties remain hidden. This approach definitely favors those people who are comfortable initiating contact with a potential match, and those who want a more “hands-on” approach to dating rather than leaving more of the process to a computer system that dictates a prescribed number of stages that need to be completed before more open communication can occur.
In contrast, eHarmony strongly recommends its members complete a prescribed set of information exchanges before “open” unrestricted communication is made available. This approach can take several weeks to complete, which for some users works well, but for many eHarmony users this was cited as one of the most significant drawbacks of the site. For online daters who are unsure of themselves or shy, the availability of multiple choice questions and answers, and lists of “must/haves and can’t stands” are attractive. This is because early exchanges with a potential match do not require users to stare at blank email forms and agonize over what to write. However, a more seasoned dater summed up the approach as “dating training wheels” given (a) the limitations that these tools impose on the ability to truly express one’s personality, and (b) the delay created in being able to set up a date if it’s felt compatibility may be present.
One of the users (who we would describe as “very seasoned”) provided a very interesting insight into how he prioritizes his communications with potential dates. First, he classifies messages (emails, winks etc) as either “outbound” or “inbound.” Outbound communication is initiated by him whereas inbound connections are not. He feels that this is an important distinction as his outbound messages (and responses) are much more valuable to him than his inbound communication, which, while flattering, may be from potential dates he is not interested in. Whereas, all of his outbound communication targets potential dates he definitely wishes to explore further.
This is an excerpt from a February 2010 pdf report.