You‘re more likely to encounter problem participants in a remote unmoderated study, as compared to remote moderated or in-person usability testing studies — especially if you recruit from panels hosted by dedicated testing services.

It’s important to identify people whose behavior is not representative for your user population and exclude their data from your analysis. (Testing representative users is one of the可用性测试的核心原则那and unrepresentative participants invalidate many of the findings from a study.)

In this article, we’ll discuss how to identify three types of problem participants: outliers, cheaters, and professional participants.

异常值s是participants whose behavior or performance is very different from the rest of your user population, either because they are not part of your target user group or because they are in exceptional in some other way.

骗子participants interested only in getting paid and moving on to the next study. They may click randomly and not even attempt to perform the tasks.

Professional participants是人们参加太多研究的人。通常,这些人不代表'普通'用户,因为他们已经看到了太多的UX研究研究,并且过于研究人员的目标。

Problem participants can be professional participants, cheaters, or outliers.

Note thatcheaters are often outliers— in other words, people rushing through the test without really trying usually stand out from other participants in your data in some way.

However,not all outliers are cheaters。有些用户将与您的其他人不同,因为他们不同,不是因为他们试图欺骗你的激励钱。(在过去的研究中,我们发现了6% of task attempts were uncommonly slow那which we explained by “bad luck” since we didn’t have a better explanation for these outliers.)


Qualitative Studies: Watch the Recordings


While you watch the videos, keep an eye out for signals that you might have a problem participant.


看for any comments or behaviors that could tell you that theparticipant has a different experience level, background, or motivation从您的其余用户。



Look for participants whodon’t try the tasksat all. Sometimes you’ll even see participants who receive the task instructions, don’t read them, and then go off and read Facebook or some other site for a few minutes, before clicking to advance to the next task.

However, just because someone is与您的设计不耐烦不会让她成为骗子。Many users are demanding and expect products to work perfectly and easily on the first try. If your网站永远需要加载那they may try to do something else meanwhile. There’s a difference between an impatient, demanding user and one who does not attempt to perform the task at all.

Another signal for a cheater is that your participant ignores the task instructions or parameters.


Professional-Participant Signals


I tend to identify professional participants more easily by their comments than by their behaviors. Listen forany terminology that betrays too much knowledgethat they might have picked up from participating in studies too frequently (“SEO,” “kerning,” “mental model,”, “menu bar,” “hamburger”).




If you ran a quantitative study, with more than 30 participants, watching all of the videos may not be practical. You can use metrics to help you decide which videos to check. You can also spot-check each video (by watching several minutes of each one).

Most quantitative usability studies involve collecting at least two common metrics: time on task and task success.Remote unmoderated testing tools通常当t自动收集这两个指标hey run the test, so you probably already have access to them. Look at these metrics to identify responses outside of the normal range of your data.Check multiple metrics为了帮助您决定个人参与者看起来可疑,然后观看视频for those participants to confirm that they are indeed nonrepresentative.


Time on Task

查看各个任务的任务时间的频率分布,以及所有参与者的总会话时间,以识别这些移动更快或更慢than the rest of the participants.

The two participants who completed the task in less than 9 seconds were much faster than the rest. These might be cheaters. The four participants who completed the task in more than 179 seconds could be cheaters, unrepresentative participants, or just people who needed more time to complete the task. You’ll have to investigate to find out. (The histogram shows, for each time interval, the number of participants whose time on task was in that interval.)

When participants complete tasks and sessions very quickly, they might be cheaters. When they complete tasks and sessions much more slowly than the other participants, it’s possible that they’re cheaters, outliers, or neither — just people who need a little extra time or encountered an error.

Task Success

同样,我们可以通过整个会议的每个参与者的参与者查看任务成功。Low success rates combined with very fast task times是usually a strong indicator of a cheater.



计划定量研究时,包括包括的好主意一个问题与开放式文本字段,where participants have to type a response. You can quickly scan the list of responses and identify any “lazy” answers that might signal a cheater.

Let’s look at some real-life responses to the open-ended question “If you could change something about this website, what would it be?”

Example Response






“Its fine”



“On the main page I would put more basic, compelling information about the ocean -- everyone [almost] has some connection to it, whether it be the beach as a kid, trade, kayaking, swimming, cruises, boats, etc. I would just stress if possible the amazing ocean animals/life and how important the ocean is to trade, military, fun etc.”

Detailed, thoughtful responses like this one are a strong signal that this is not a cheater participant.

Next Steps





However, be sure that the participant truly isn’t representative of your user population. A UX professional once asked me if he could remove one participant’s data from his analysis, because she had given a very negative response while other participants were positive. I asked, “Well, do you have any other reason to believe she’s somehow unrepresentative?” He did not. An unfavorable response to our design is not a good enough reason to remove someone from the data.


It’s often the case that a participant might “cheat” on one or two tasks, particularly at the end of a long session, but will actually try the others. Determine whether that’s the case with your cheater, by watching the participant’s full video. If the participant cheated only on one or two tasks, simply remove her data for that task. If they cheated through the whole session, remove them entirely.

Most remote testing tools are aware that cheaters are a problem. If you recruited the participants through the tool’s panel, many offer to replace cheater participants for free if you request it.



Professional participants are sometimes trickier to deal with. In most cases, they haven’t done anything wrong — they showed up and participated, so they should be compensated and not reported or negatively reviewed.

The best thing is to avoid letting these people into your study in the first place. I always include a screener question that asks how recently the respondent participated in a study, and I exclude those who participated too recently (0–3 months or 0–6 months). However, there’s nothing to stop participants from lying in response to that question. Some testing tools allow you to filter out frequent participants automatically. However, these professionals are often performing tests on many different platforms (and there are a lot out there).



Keep any eye out for outliers, cheaters, and professional participants in your studies. Investigate multiple sources of information to help you sleuth out the situation, and decide what to do about it. If you frequently find too many problem participants in your studies, you should reevaluate how and where you recruit participants.

For more help planning, conducting, and analyzing remote studies, check out our full-day seminar,远程可用性测试。