In remote usability studies, it's hard to identify test participants who should not be in the study because they don't fit the profile or don't attempt the task seriously. This is even harder in unmoderated studies, but it can (and should) be done.
To learn something useful from a usability study, you must have the test participants perform tasks that are representative of typical user goals, while avoiding bias caused by giving too detailed directions or hints.
There are two ways to structure a UX research study when we're testing two (or more) designs: we can have each design tested by different people, or we can reuse the same users for all conditions. Each approach has some advantages and problems.
For 30 years, the recommendations have remained the same for improving usability in a UX design project on a tight budget: simplified user testing with 5 users, early test of paper prototypes, and heuristic evaluation.
Usability testing and other UX evaluation methods can be divided into two major categories: formative evaluation and summative evaluation. Both have their place, but at different stages in the design lifecycle, and they have different characteristics, for example in the number of test participants needed for a good study.
Usability testing is similar to how wild animals hunt for food: we're trying to hunt down the design flaws in the user interface and must optimize a series of studies for total gain, rather than spend too much on any one study.
“That’s just one person” and “Our real users aren’t like that” are common objections to findings from qualitative usability testing. Address these concerns proactively to ensure your research is effective.