Research design: online chats vs. face-to-face group discussions

amysantee, ,

There is a lot of talk these days about online qualitative research. I recall a meeting from late last year during which some coworkers discussed the possible advantages of online research over traditional methods like group discussions (aka focus groups) and in-depth interviews. They were particularly interested in the potential to save time and money on things like participant incentives, researcher travel, and facility/moderator costs.

I think that there’s a time and place for online research, and I don’t think it can or should always be used as a substitute for in-person research. Resource and time permitting, face-to-face groups (and better yet, ethnographic or observational research if appropriate) seem to be better options because of their potential to yield deeper insights from consumers by providing an unparalleled setting for group dynamics and social interaction. I do think that online research can make a good partner in a mixed-methods project, and can work if you’re in a pinch or don’t have a lot of money. Decisions about research design should be based on the question or problem at hand and the monetary and human resources that are available.

One example of an online research method that’s growing in popularity is moderated chats. Much like face-to-face discussions, online chats involve recruiting a group of consumers, users or stakeholders that meet specific criteria (demographics, customer segments, etc.) to participate in a moderated, semi-structured chat around a particular topic. The goal, as with face-to-face groups, is to gain insights that are more in-depth than what can be gleaned from a survey (they can also be used to compliment or develop a survey), but in a shorter amount of time than would be needed to interview everyone individually. If possible, take the opportunity to learn more about incorporating online research into the context of your organization’s research efforts to see if it makes sense, but be aware of the potential effects of cost-cutting on the quality of data.

There are methodological, logistical and financial benefits and limitations to every research tool that exists, and it’s important that we recognize them. I don’t personally do much online research these days, but I thought it might be useful to put together a side-by-side comparison of both methods for those who are trying to decide which one to use. This is by no means an exclusive list, but this is what I could glean from a few sources I found online and in my bookshelves (listed below). Likewise, some of what is listed here may also pertain to a comparison of other, similar qualitative methods (like in-depth interviews versus Skype interviews, for example).

Considerations for Online Chats vs. F2F Groups
Issue
Online Chats
F2F Groups
Timeline
BENEFITS: great for projects that need quick turnaround time (a week or so)
LIMITATIONS: quick turnaround might not yield best data and limits depth of research
BENEFITS: quicker than some other methods (e.g., in-depth interviews)
LIMITATIONS: usually takes a day to do 2-3 groups in one city; travel to other cities to get geographic diversity  = a few weeks or more from start to finish
Recruiting
BENEFITS: ability to screen participants for selection criteria; people can be anywhere at any time; may increase access to hard-to-reach populations (e.g., doctors, busy professionals) and special interest groups; potentially cheaper recruit costs; may use online gift cards (e.g., Amazon) for incentives instead of cash; also, some companies have private online communities with a reserve of people waiting to be contacted for this type of research
LIMITATIONS: if you can’t see someone, you don’t know if they are who they say they are; possible selection bias (e.g., skewed toward the internet-savvy)
BENEFITS: ability to screen participants for selection criteria
LIMITATIONS: participants have to be in one location; might skew toward stay-at-home parents, retirees, unemployed, students unless selection quotas are used; potentially higher recruit costs, and most participants prefer cash-in-hand
Participant dynamics
BENEFITS: Dominant talkers, shy participants and ramblers less of an issue; anonymity lessens “social desirability” effect and might facilitate positive social interaction
LIMITATIONS: Loss of face-to-face group dynamics and richness of social interaction leading to disconnected participants; anonymity might facilitate negative social interaction; potential for group think
BENEFITS: Rich group dynamics, visual body language, face to face social interaction
LIMITATIONS: Dominant talkers taking over, shy participants not participating, ramblers; participants might engage in “group think” and refrain from sharing true opinions because of how they may be perceived by others  
Level of participation
BENEFITS: participants can do it on their own time and from the comfort of their own home or office, or a coffee shop
LIMITATIONS: multitasking might get in the way of full participation = half-hearted participation
BENEFITS: participants possibly more compelled to participate since they’re there
LIMITATIONS: participants get bored, their minds go elsewhere if it’s a boring topic, if they’re tired, if they have something on their minds, if the moderator isn’t effective
Individual contributions/talk time
BENEFITS: participants may contribute at any time without waiting for others to finish; no need to take turns or worry about long responses or rambling; potential for more equal distribution of participation
LIMITATIONS: potentially slower interactions and pacing
BENEFITS: richness of discussion facilitated by pro moderator
LIMITATIONS: limited amount of time for everyone to share; moderator has to work to maintain pacing, control participants, incite participation  
Number of participants
BENEFITS: moderator can handle more participants without being overwhelmed; can follow-up with multiple people at once
LIMITATIONS: barrage of responses might cause missed opportunity to ask follow-up questions, especially if there are too many participants
BENEFITS: 6 or 7 people (max) per group allows for everyone to contribute at a deeper level; fewer people to keep track of/respond to
LIMITATIONS: only one person at a time can talk
Client viewers
BENEFITS: a potentially unlimited number of clients can tune-in on the conversation and send questions to moderator via email, chat, conference call or in-person; transcripts available immediately; presence of viewers potentially more easily forgotten than in focus group
LIMITATIONS: not as interesting as viewing a real group of people; clients may  be distracted by other work; flat text is not as interesting as live conversation
BENEFITS: in traditional market research facilities, clients can view the research live from behind one-way mirror or in back of room
LIMITATIONS: moderator must leave room to get client questions, or someone has to send in a note which interrupts discussion; participants may feel uncomfortable knowing they’re being watched and being reminded by one-way mirror
Moderating
BENEFITS: potentially easier to moderate because of limited dynamics; can have multiple moderators (e.g., one can lead while the others probe) to keep things moving
LIMITATIONS: have to be able to keep up with text-based conversation; lack of body language disallows moderator from using body language to his/her advantage; the chat might turn out to be a serial interview rather than an actual discussion
BENEFITS: skilled moderators can take on the dynamics, context and topic of any group and elicit insightful information; easier to facilitate a “discussion” rather than simply asking a battery of questions one-by-one
LIMITATIONS: human dynamics always make things more complicated, so moderator has to be good at controlling them
Follow-up questions/probing
BENEFITS: can follow up with multiple people at once, individually or as a group
LIMITATIONS: limited use of probing techniques (e.g., can’t use probes that involve silence, body language)
BENEFITS: numerous techniques for probing involving language, body language, silence, etc.
LIMITATIONS: following up with individuals may be easier because only one person can speak at a time
Stimuli
BENEFITS: ability to show text, video, images without special equipment; people can react on their own
LIMITATIONS: limited options (e.g., no physical prototypes or anything that can’t be digitized)
BENEFITS: numerous opportunities to incorporate stimuli like text, video, images, prototypes, social interactions, etc.
LIMITATIONS: might take more time to set up or need special equipment
Location
BENEFITS: convenient for participants, researchers/moderator, clients (everybody can stay where they are); no fees for facility rental or travel
LIMITATIONS: everyone being in different places could potentially create some minor inconveniences
BENEFITS: nice to have everyone in one place to facilitate discussions and benefit from face-to-face interaction
LIMITATIONS: it costs money! (researcher/moderator travel, client travel, facility rental, and higher incentives for participants)
Cost
BENEFITS: overall way cheaper; incentives for participants can be lower since they don’t have to show up in person and because chats are usually shorter in duration
LIMITATIONS: you get what you pay for
BENEFITS: you get what you pay for
LIMITATIONS: overall way more expensive
Output (transcripts, video/audio) and data analysis
BENEFITS: quick data collection; immediate transcript from chat record; no need for transcribing; can start data analysis right away
LIMITATIONS: discussion may be less interesting, interactive and in-depth because of lack of human dynamics; no video or audio recordings available, just flat text; data = less robust?
BENEFITS: verbatim transcripts and video illustrate a real, face-to-face conversation; video/audio can be reviewed; data = more robust?
LIMITATIONS: transcripts take longer because videos/audio must be transcribed by a person (which may take days/weeks)
References:

Share your thoughts