I keep hearing the same response whenever I suggest running an A/B test on a new product or a lightly used feature.
“We do not have enough traffic for statistical significance.”
Usually, that is true. If 40 people use a product each week, splitting them between two versions will produce two tiny piles of noisy data. The team can wait months and still learn very little about a modest change in conversion.
If a team has enough traffic for a proper A/B test, instrument it properly. I still want to watch the session recordings before I open the dashboard because going in blind keeps the result from telling me what to see. After I write down where people hesitate or get sent on a detour, I compare those notes with the quantitative result.
If I only have 40 sessions, I am watching all 40. Most of the useful data is qualitative at that point because the sample cannot support the claim no matter how carefully I build the measurement system. When a recording leaves an important question open, I can hopefully talk to the person.
Numbers are good at winning arguments
Experiment results travel well inside an organization. A chart gives people something solid to put in a deck. A percentage can make a judgment look settled, which helps its owner defend a proposal in a meeting.
A clean dashboard result is easy to carry into the meeting, so the recordings get skipped and useful behavior disappears behind the outcome metric.
An event is a choice somebody encoded. They chose when it fires and which sessions count. Those definitions can be bent during an argument, even when everyone involved believes they are being reasonable. Honest instrumentation has a narrower problem too: it only captures behavior the team thought to name in advance.
A funnel can tell you that five people left on the second step. The recording shows one of them clicking a disabled button four times before scrolling back to find a requirement that was never explained. When the page fails again, they leave. The event history compresses that visit into an exit.
The recording still cannot tell you why the visitor acted that way. Ask them when the reason matters. For most early interface problems, seeing what the product did in response to a person is enough to get started.
Funnels and events measure the outcome and point you toward sessions worth watching. Watch the recordings first, then use the funnel to compare the outcome with the behavior you saw.
Five users can give you plenty to fix
Laura Faulkner tested 60 people for a 2003 usability study, then repeatedly drew random groups of different sizes from that full set. The five-person groups found anywhere from 55 percent to 99 percent of the known usability problems. Among the groups of 20, even the worst-performing group found 95 percent.
Five is a useful starting point for finding problems. In Faulkner’s study, one group of five saw nearly everything and another missed almost half, so a few clean sessions cannot certify the interface. They can still fill your next week with useful work.
Early product teams get twisted around by measurement language when they ask a discovery method to estimate prevalence, then dismiss it for failing to produce that estimate. The first question is usually much more practical: what is going wrong here, and what should I fix next?
Session recordings are unusually good for that question because low traffic becomes an advantage. You can watch every relevant visit. There is no sampling plan to design and no dashboard to maintain before you learn anything. Open the recordings, take plain notes about what happened on screen, and look for the places where the product repeatedly lets people down.
Let the product carry the argument
Use this research as a tool for making the product better. Once its main job becomes convincing other people, the incentives pull the work toward harder-looking evidence and more elaborate methods. The work gets easier to present as it becomes less useful for the product in front of you.
Some projects deserve the whole measurement project. A change that is expensive or painful to reverse needs a claim defined in advance and enough evidence to support it.
Most early product work is cheaper to correct, so watching later sessions belongs in the change itself.
This way of working is less visible. You will have fewer charts for leadership and fewer impressive numbers for peers to repeat. Some of the value will live in your judgment, which makes it harder to claim as a neat win.
When a dead control works after the fix, users stop taking the detour you saw in six recordings. They get a better product, even if nobody in the meeting knows why.