One of the enduring controversies and sources of confusion in UX research concerns sample size.<\/p>\n Part of the reason for the confusion is that there are different perspectives; some are more vocal than others. This isn\u2019t different from other enduring controversies, such as between frequentist and Bayesian statisticians<\/a>.<\/p>\n Even among frequentist statisticians, there\u2019s been a long debate about how to interpret the p<\/em>-value<\/a> as either a continuous measure of evidence (Fisher) or as a cutoff for significance decisions (Neyman-Pearson). In fact, debates and philosophical disagreements exist in most scientific fields (such as dark matter and dark energy in cosmology<\/a> or adaptationism vs. structuralism<\/a> in evolution). And, of course, one of the original philosophical debates<\/a> was between Plato\u2019s emphasis on ideals and reason versus Aristotle\u2019s focus on empiricism.<\/p>\n We\u2019ve noticed there seem to be two schools of thought when discussing sample sizes in UX. Both are too extreme, but both have kernels (or more) of truth. One school preaches that small sample sizes are always enough; the other, that you always need very large sample sizes.<\/p>\n As is often the case with vocal opinions, the truth lies somewhere in between. First, let\u2019s understand the philosophies of each school.<\/p>\n If this school had a motto, it would be \u201cFive is just fine.\u201d<\/p>\n The first school of thought is that small sample sizes are more than enough. This school believes that a few users suffice for just about any UX research study, from uncovering problems, determining preferences, interviewing prospective users, or assessing a new design\u2019s effectiveness.<\/p>\n Also common with this school of thought is the prohibition on quantification. No numbers, no questions about statistical significance.<\/p>\n Five is always enough. Regardless of the method or objective, a few people are all you need. More than five is a waste of resources. Five will provide saturation, and five will prove preferences.<\/p>\n If this school had a motto, it would be \u201cLarge is in charge. Five is never enough.\u201d<\/p>\n In this school, five is a rounding error. When there is a five in the sample size, it\u2019s usually followed by two or three zeros.<\/p>\n This school believes that strong statements about generalizations (what users prefer, which design is better) require large sample sizes because using statistics requires large sample sizes.<\/p>\n Five users are meaningless. This school would also agree that you can\u2019t quantify data from sample sizes of five. And you certainly can\u2019t use statistics on small samples. Using large sample sizes allows you to use statistics and brings respect to the field.<\/p>\n But there is another school, one we invite you to join.<\/p>\n The third school on sample size is less dogmatic and more pragmatic. In this school, the right sample size is based on objectives,<\/em> not objections<\/em>. Sometimes five is fine, and sometimes it\u2019s abysmally too small. The pragmatic school of thought is not based on picking and choosing what sounds good. It\u2019s based on starting from the goal and defining the approach that works best mathematically in the long term\u2014let\u2019s say starting with Plato and applying Aristotle.<\/p>\n We recommend and teach that finding the right sample size starts with the research goal, and the optimal size will vary. UX research goals typically fall into one of three approaches.<\/p>\n One research goal is to discover problems in an interface or glean insights from interviews or real-world observations. Here, the sample size is based on the expected frequency of the unknown problem\/insight<\/a> (or a range of possible frequencies) and how high a chance you need to uncover it. Typically, when a serious problem is discovered once (like tripping on a carpet<\/a> or other safety issue), you fix it and don\u2019t need to see it again. But the likelihood of discovering a critical issue, like any other issue, depends on the frequency with which it will happen<\/a>. Problems that impact many people (e.g., more than 30% of the population) can be detected with as few as five users. Less common problems (e.g., those that affect 10% or less of the population) require larger sample sizes (20+) to detect.<\/p>\n Population parameters include the average completion rate, average rating scale score, average NPS, median time, or association\/correlation between metrics. This method uses a confidence interval, and sample size strongly influences the interval\u2019s margin of error. Precision and sample size follow an inverse square relationship<\/a>. To cut the margin of error in half (e.g., 20% to 10% margin of error), you need to roughly quadruple the sample size. Interestingly, this is the same law that governs why you need four times the transmission power to double the distance a signal needs to travel<\/a>. The sample size you need for estimating population parameters depends on three things<\/a>: the variability of the measurement, the desired confidence level, and the desired precision of measurement (margin of error).<\/p>\n Is one design rated as easier than another? Which design do users prefer? Is the conversion rate higher with the new design than with the older one? Did the NPS change this quarter compared to last quarter? This sample size is driven largely by the same factors that matter when estimating parameters (variability, confidence, and error limit) plus two additional factors<\/a>: power and experimental design (between- or within-subjects). The conception of error limit is similar but not exactly the same for estimation and comparison. Estimation focuses on controlling the margin of error around a measurement, while comparison focuses on controlling the minimum size of the difference (the critical difference) between two measurements (although this can also be conceptualized as controlling the margin of error around a difference). Also, like estimation, sample sizes for comparisons follow the inverse square law (to cut the critical difference in half requires quadrupling the sample size).<\/p>\n <\/p>\n Table 1:<\/strong> Positions of the two absolute schools and the pragmatic school on three research practices.<\/p>\n Absolute claims about sample sizes and related research practices are easy to follow but are often wrong. When it comes to sample size estimation, we recommend pragmatic practices that are consistent with the math that underlies the relevant research goal (e.g., discovery, estimation, comparison).<\/p>\n<\/p><\/div>\n\n","protected":false},"excerpt":{"rendered":" Five users are enough. Or do you need a large sample size to make statistically significant claims? One of the enduring controversies and sources of confusion in UX research concerns sample size. Part of the reason for the confusion is that there are different perspectives; some are more vocal than others. This isn\u2019t different from […]<\/p>\n","protected":false},"author":1,"featured_media":300059,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[12033],"tags":[14041,10113,13861,13732,782,4821],"dealstore":[],"offerexpiration":[],"class_list":["post-300058","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-analytics","tag-measuringu","tag-research","tag-sample","tag-schools","tag-sizes","tag-thought"],"yoast_head":"\n![\"Feature](\"https:\/\/measuringu.com\/wp-content\/uploads\/2025\/06\/061725-FeatureImage.png\")
<\/a>Five users are enough. Or do you need a large sample size to make statistically significant claims?<\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\nFinding Problems\/Insights<\/h2>\n
Estimating a Population Parameter<\/h2>\n
Making Comparisons<\/h2>\n
\n\n
\n \nResearch Practice<\/th>\n Small Sample School<\/th>\n Large Sample School<\/th>\n Pragmatic School<\/th>\n<\/tr>\n<\/thead>\n \n Sample size<\/td>\n Small n<\/i> always good enough<\/td>\n Small n<\/i> never good enough<\/td>\n Low to high n<\/i> depends on objective<\/td>\n<\/tr>\n \n Quantification<\/td>\n Never report numbers<\/td>\n Always report numbers<\/td>\n Numbers often help even at small sample sizes<\/td>\n<\/tr>\n \n Assess statistical significance on small sample sizes<\/td>\n Don\u2019t do it<\/td>\n Don\u2019t do it<\/td>\n Small samples limit what stats you can detect and project but they don\u2019t prohibit stats usage<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n