When you want to compare several user interfaces in a single study, there are two ways of assigning your test participants to these multiple conditions:

  • 科学之间(or之间的团体) study design: different people test each condition, so that each person is only exposed to a single user interface.
  • 在受试者内(或重复措施)研究设计:同一个人测试所有的管道ions (i.e., all the user interfaces).

(请注意,这里我们使用“设计”一词来引用design of the experiment, and not to website design.)

For example, if we wanted to compare two car-rental sites A and B by looking at how participants book cars on each site, our study could be designed in two different ways, both perfectly legitimate:

  1. 科学之间: Each participant could test a single car-rental site and book a car only on that site.
  2. Within-subjects: Each participant could test both car-rental sites and book a car on each.


Experimental Design in Quantitative Studies

Unlike qualitative studies,quantitative usability studies旨在导致对整个用户群体概括的调查结果。如何分析这些研究的数据如何取决于设计设计的方式(即,在研究中experimental design)。

通常,定量可用性研究的主要目标是比较 - 与其竞争对手的网站,两个不同的设计迭代,或两组不同的用户组(例如专家与新手)。喜欢在我们想要检测因果关系的任何科学实验中,定量研究涉及两种类型的变量:

  • 独立变量, which are directly manipulated by the researcher
  • 因变量, which are measured (and expected to vary as a result of the independent-variable manipulation)

(If the study produces statistically significant results, then we can say that a change in the independent variablecauseda change in the dependent variable.)

Let’s go back to our original car-rental example. If we wanted to measure which of the two sites, A or B, is better for the task of reserving a car, we could choose现场(with two possible values orlevels- a和b)作为独立变量,以及任务的时间和预订汽车的准确性可能是依赖变量。该研究的目标是看看当我们改变网站或保持不变时,是否有所依赖变量(时间和准确性)更改。(如果它们保持不变,那么这些网站都比另一个网站更好。)

To plan our study, the next question that we need to answer is whether the study design should be between-subjects or within-subjects — that is, whether a participant in the study should be exposed to all the different conditions for the independent variable in our study (within-subjects) or only to one condition (between-subjects). The choice of experimental design will affect the type of statistical analysis that should be used on your data.

It is possible that an experiment design is both within-subjects and between-subjects. For example, assume that, in the case of our car-rental study, we were also interested in knowing how participants younger than 30 perform compared with older participants. In this case we would have two independent variables:

  • 年龄, with 2 levels: under 30, over 30
  • 现场,2级:a和b

For the study, we will recruit an equal number of participants in each age group. Let’s assume that we decide that each participant, whether under or over 30, will make a car-rental reservation on both site A and on site B. In this case, the study is within-subjects with respect to the independent variable现场(因为每个人看到这个变量的两个级别 - 即站点A和网站B)。然而,研究是对科学之间的主题年龄:一个人只能在一个年龄组(或超过30,而不是两者之间)。(嗯,从技术上讲,您可以选择一组30岁的孩子,等待他们30岁,让他们再次测试这些网站,但对于大多数现实世界的情况来说,这个设置非常不切实际。)

一些独立的变量可能会施加设计的选择。年龄is one of them, as seen above. Others areExpertise(if we want to compare experts and novices),User Type(如果我们想比较不同的用户组或角色 - 例如,商务旅行者与休闲旅行者),或Gender(假设一个人同时不能是几个性别的)。外部可用性,药物试验是对象设计的一个常见情况:参与者仅暴露一次治疗:要么被测试的药物或安慰剂,而不是两者。And sometimes the manipulation itself changes the state of the participant: for example, if you want to see which of two curricula is more effective for teaching reading, you could not have the same student be exposed to both, because once she’s learned how to read, she cannot unlearn it.

Which Is Better: Between-Subjects or Within-Subjects?

Unfortunately, there is no easy answer to this question. As seen above, sometimes your independent variables will dictate the experimental design. But in many situations, both designs may be possible.

  • 科学之间minimizes the learning and transfer across conditions.在一个人在汽车租赁网站上完成了一系列任务后,她更了解域名比她以前的域名。例如,她现在可以知道汽车租赁网站为21岁以下的司机收取额外费用,或者碰撞损害是什么。这种知识可能有助于她在第二个汽车租赁网站上变得更加高效,尽管第二个网站可能与第一个站点非常不同。

    With between-subject design, this transfer of knowledge is not an issue — participants are never exposed to several levels of the same independent variable.

  • 受试者之间的研究比在主题内的较短时期。测试单个汽车租赁网站的参与者将与测试两个人的会话较短。较短的会话对用户疲软(或无聊)较少,并且也可以更适合远程未经寄存的测试(特别是因为用户zoom等工具通常需要相当短的会话长度)。
  • 对象之间的实验更容易设置,特别是当您有多个独立变量时。When the study is within-subjects, you will have to use randomization of your stimuli to make sure that there are no order effects. For example, in our car-rental study, we need to make sure that participants don’t always start with site A and then move on to site B. The order of the sites needs to be random for each participant. This is easy with just two sites: randomly assign 50% of users to start with each site. But as you increase the number of independent variables and of levels for an independent variable, randomization becomes more difficult to implement within some of the existing platforms for quantitative usability testing.
  • Within-subject designs require fewer participants and are cheaper to run.To detect a statistically significant difference between two conditions, you’ll often need a fair number of a data points (often above 30) in each condition. If you have a within-subject design, each participant will provide a data point for each level of the independent variable. For our car-rental study, 30 participants will provide data points for both sites. But if the study is between-subjects you will need twice as many to get the same number of data points. That means twice the cost.
  • Within-subjects design minimize the random noise.也许受试者内部设计中最重要的优势在于它们使得您的条件之间存在的实际差异不太可能会保持未检测到或被随机噪声覆盖。

    Individual participants bring in to the test their own history, background knowledge, and context. One may be tired after a long night of partying, another one may be bored, yet another one may have received a great news just before the study and be happy. If the same participant interacts with all levels of a variable, she will affect them in the same way. The happy person will be happy on both sites, the tired one will be tired on both. But if the study is between-subjects, the happy participant will only interact with one site and may affect the final results. You’ll have to make sure you get a similar happy participant in the other group to counteract her effects.

    在实践中,研究人员将无法评估uch differences between participants — although they may match the gender, the experience, and the age across groups, it will be difficult to predict or detect other factors specific to each participant.


Whether your experimental design is within-subjects or between-subjects, you will have to be concerned with randomization, although in slightly different ways.


For between-subject designs, you must make sure that participants are allotted randomly to conditions, because you want to ensure that your participant assignment does not affect your study results. Thus, if a researcher decides that all the participants that he likes should interact with site A and then he finds that site A performed better than site B, he won’t know whether he’s discovered a true difference between the sites or whether the result simply reflects his assignment (for example, because people who sense that they are liked tend to return the favor, and may be more patient or have a positive mindset during the test).

即使没有如此明显的偏见作为您的个人喜好,也很容易让随机化错了。假设您在周二周二的四天内进行学习。You might decide to have the first half of the test users start with site A and have the second half of the users start with site B. However, this is not a true randomization, because it’s very likely that certain types of people are more likely to agree to a study during the weekend and other types of people are more likely to sign up for your weekday testing slots.