You‘re more likely to encounter problem participants in a remote unmoderated study, as compared to remote moderated or in-person usability testing studies — especially if you recruit from panels hosted by dedicated testing services.

It’s important to identify people whose behavior is not representative for your user population and exclude their data from your analysis. (Testing representative users is one of the可用性测试的核心原则那and unrepresentative participants invalidate many of the findings from a study.)

In this article, we’ll discuss how to identify three types of problem participants: outliers, cheaters, and professional participants.

异常值s是participants whose behavior or performance is very different from the rest of your user population, either because they are not part of your target user group or because they are in exceptional in some other way.

骗子participants interested only in getting paid and moving on to the next study. They may click randomly and not even attempt to perform the tasks.

Professional participants是人们参加太多研究的人。通常,这些人不代表'普通'用户,因为他们已经看到了太多的UX研究研究,并且过于研究人员的目标。

Problem participants can be professional participants, cheaters, or outliers.

Note thatcheaters are often outliers— in other words, people rushing through the test without really trying usually stand out from other participants in your data in some way.

However,not all outliers are cheaters。有些用户将与您的其他人不同,因为他们不同,不是因为他们试图欺骗你的激励钱。(在过去的研究中,我们发现了6% of task attempts were uncommonly slow那which we explained by “bad luck” since we didn’t have a better explanation for these outliers.)

识别问题参与者的最佳方法取决于您是否正在运行定性或定量可用性测试

Qualitative Studies: Watch the Recordings

大多数远程未经寄存的测试平台在执行任务时记录参与者屏幕的视频(有时是他们的网络摄像头)。如果你正在运行一个定性测试5-8名参与者然后,您应该计划在分析的一部分中观看所有视频。

While you watch the videos, keep an eye out for signals that you might have a problem participant.

异常信号

看for any comments or behaviors that could tell you that theparticipant has a different experience level, background, or motivation从您的其余用户。

例如,如果您招聘工业工程师,但一个参与者声音对UI中使用的术语非常困惑,但他实际上可能在这一领域中实际上没有背景。如果您没有在筛选器中询问正确的问题以评估知识,他可能是一个刚刚在错误的研究中最终结束的良好意图的参与者。

骗子信号

Look for participants whodon’t try the tasksat all. Sometimes you’ll even see participants who receive the task instructions, don’t read them, and then go off and read Facebook or some other site for a few minutes, before clicking to advance to the next task.

However, just because someone is与您的设计不耐烦不会让她成为骗子。Many users are demanding and expect products to work perfectly and easily on the first try. If your网站永远需要加载那they may try to do something else meanwhile. There’s a difference between an impatient, demanding user and one who does not attempt to perform the task at all.

Another signal for a cheater is that your participant ignores the task instructions or parameters.

例如,如果任务是“访问西榆树,并找到一个带有金属框架的餐椅,”,但参与者选择了他所看到的第一个项目(一张咖啡桌)(并说他所做的),然后他可能只是努力尽快离开任务。

Professional-Participant Signals

专业参与者是最难以识别的。这些是(大多数情况下)的人试试您的任务并为您提供大量反馈。他们往往很擅长大声思考。他们做了很多研究,他们已经了解了研究人员想要的东西。

I tend to identify professional participants more easily by their comments than by their behaviors. Listen forany terminology that betrays too much knowledgethat they might have picked up from participating in studies too frequently (“SEO,” “kerning,” “mental model,”, “menu bar,” “hamburger”).

(注意:这些天很多人都学到了“用户友好”,“可用性”的条款,甚至是来自广告和流行文化的“用户体验”,所以这并不总是警告标志。)

例如,在搜索非营利性网站上的信息时,我的一名学习参与者表示,“看起来,使用搜索引擎的魔法逃脱了很多人。他们无法形成查询使他们认为搜索引擎是无用的。在我的经验中,学习如何改写你的问题,以便您正在寻找的答案对他们的使用至关重要。所以它并不是很容易或困难,它更让用户对搜索功能的经历。“正确使用的术语的数量让我怀疑这个人可能是专业参与者,或者可能已经在UX相关领域工作。

定量研究:以指标开头

If you ran a quantitative study, with more than 30 participants, watching all of the videos may not be practical. You can use metrics to help you decide which videos to check. You can also spot-check each video (by watching several minutes of each one).

Most quantitative usability studies involve collecting at least two common metrics: time on task and task success.Remote unmoderated testing tools通常当t自动收集这两个指标hey run the test, so you probably already have access to them. Look at these metrics to identify responses outside of the normal range of your data.Check multiple metrics为了帮助您决定个人参与者看起来可疑,然后观看视频for those participants to confirm that they are indeed nonrepresentative.

请注意,基于度量标准的方法对于识别异常值和欺骗,而且通常对于识别专业参与者通常是有用的。

Time on Task

查看各个任务的任务时间的频率分布,以及所有参与者的总会话时间,以识别这些移动更快或更慢than the rest of the participants.

The two participants who completed the task in less than 9 seconds were much faster than the rest. These might be cheaters. The four participants who completed the task in more than 179 seconds could be cheaters, unrepresentative participants, or just people who needed more time to complete the task. You’ll have to investigate to find out. (The histogram shows, for each time interval, the number of participants whose time on task was in that interval.)

When participants complete tasks and sessions very quickly, they might be cheaters. When they complete tasks and sessions much more slowly than the other participants, it’s possible that they’re cheaters, outliers, or neither — just people who need a little extra time or encountered an error.

Task Success

同样,我们可以通过整个会议的每个参与者的参与者查看任务成功。Low success rates combined with very fast task times是usually a strong indicator of a cheater.

在不到9秒内完成任务5的两位参与者在所有任务中都有非常低的成功率。旗帜这些参与者并通过观看视频来跟进。(直方图显示了在X轴上显示的间隔的成功率的参与者的数量。)

开放式文本响应

计划定量研究时,包括包括的好主意一个问题与开放式文本字段,where participants have to type a response. You can quickly scan the list of responses and identify any “lazy” answers that might signal a cheater.

Let’s look at some real-life responses to the open-ended question “If you could change something about this website, what would it be?”

Example Response

Description

坏

“asiojdfoiwejfoiasjdfiasjdf”

这些废话的反应看起来像是有人刚刚抨击键盘继续进行这项研究,而不令人兴奋地阅读问题并制定回应。这是骗子的强大指标。

可疑的

“Its fine”

非常短的非答案与错误或没有标点符号,如此,有时可以用信号发出骗子,但并不总是如此。在这种情况下,参与者可能已经疲惫不堪,或者只是没有任何强烈的意见。

Good

“On the main page I would put more basic, compelling information about the ocean -- everyone [almost] has some connection to it, whether it be the beach as a kid, trade, kayaking, swimming, cruises, boats, etc. I would just stress if possible the amazing ocean animals/life and how important the ocean is to trade, military, fun etc.”

Detailed, thoughtful responses like this one are a strong signal that this is not a cheater participant.

Next Steps

当您确定问题参与者时,以略微不同的方式处理每种类型。

异常值

应完全从您的分析中删除与目标受众截然不同的异常值。

回到您的招聘,并尝试确定该人在您的学习中是如何结束的。这个人究竟是什么意思,让她不适合其余的人?您的筛选器是否存在问题,使此人进入该研究?从这个错误中学习,以确保未来更好的新兵。

However, be sure that the participant truly isn’t representative of your user population. A UX professional once asked me if he could remove one participant’s data from his analysis, because she had given a very negative response while other participants were positive. I asked, “Well, do you have any other reason to believe she’s somehow unrepresentative?” He did not. An unfavorable response to our design is not a good enough reason to remove someone from the data.

骗子

It’s often the case that a participant might “cheat” on one or two tasks, particularly at the end of a long session, but will actually try the others. Determine whether that’s the case with your cheater, by watching the participant’s full video. If the participant cheated only on one or two tasks, simply remove her data for that task. If they cheated through the whole session, remove them entirely.

Most remote testing tools are aware that cheaters are a problem. If you recruited the participants through the tool’s panel, many offer to replace cheater participants for free if you request it.

如果招聘服务有办法提供有关参与者的表现的反馈,请确保您这样做,作为对其他研究人员的礼貌。

专业参与者

Professional participants are sometimes trickier to deal with. In most cases, they haven’t done anything wrong — they showed up and participated, so they should be compensated and not reported or negatively reviewed.

The best thing is to avoid letting these people into your study in the first place. I always include a screener question that asks how recently the respondent participated in a study, and I exclude those who participated too recently (0–3 months or 0–6 months). However, there’s nothing to stop participants from lying in response to that question. Some testing tools allow you to filter out frequent participants automatically. However, these professionals are often performing tests on many different platforms (and there are a lot out there).

如果您找到了专业的参与者,请查看视频和数据以决定是否抛出会话。有时你会发现参与者制作了“专业”的发言评论,但实际的行为和数据看起来与您的其他人非常相似。在这些情况下,您可以保留数据。只需确保您的标志,参与者在您的定性分析中,权衡事实上,因为您得出结论,从该参与者的意见和反馈中得出结论。

Conclusion

Keep any eye out for outliers, cheaters, and professional participants in your studies. Investigate multiple sources of information to help you sleuth out the situation, and decide what to do about it. If you frequently find too many problem participants in your studies, you should reevaluate how and where you recruit participants.

For more help planning, conducting, and analyzing remote studies, check out our full-day seminar,远程可用性测试。