Knowledge Base / Cheat-sheets
Statistics in Testing
Sampling and coverage, describing test data with percentiles rather than averages, inference and the comparison pitfalls that produce false regressions, quality and reliability metrics, and a dedicated tooling roundup - five sections you can filter and toggle as you verify them. Progress is saved on this device.
What this checklist covers
Sampling & Coverage
Exhaustive testing is impossible, so every test run is a sample. The only question is whether it was a sample chosen deliberately or by accident - and an accidental sample supports no conclusion about anything but itself.
- The population a result is meant to describe is stated explicitly - all users, paying users, one region - because a claim with no population is unfalsifiable.
- Conclusions are not extended past the sampled population; a test over UK data says nothing certain about a locale it never touched.
- Random sampling is used when the goal is an unbiased estimate of overall quality, since any judgement-based pick inherits the tester's assumptions.
- Stratified sampling covers each meaningful subgroup - browser, region, plan tier - so a small but critical segment is not swamped by a large one.
- Sample size is derived from the smallest effect worth detecting, not from what fits the sprint.
- An underpowered test is recognised as unable to support 'no difference' - absence of evidence is being mistaken for evidence of absence.
Describing Test Data
How results are summarised decides what people conclude from them. The mean is the default summary and, for most performance and latency work, the one that hides the problem you were hired to find.
- A measure of spread accompanies every average, since a mean with no variance is compatible with wildly different user experiences.
- The median is preferred over the mean for skewed data, because a handful of 30-second requests drag the mean somewhere no user actually lives.
- Latency SLAs are set on percentiles (p95, p99) because the tail is where users abandon, and the mean is deaf to it.
- The relationship between p99 and page-level experience is understood: a page making 50 calls will hit its p99 on most loads.
- The distribution is plotted before it is summarised - a histogram routinely reveals bimodality that no summary statistic would ever show.
- Bimodal results are investigated as two populations (cache hit and miss, warm and cold start) rather than smoothed into one misleading number.
Inference & Comparison
Deciding whether a difference between two runs is real or is noise. This is where most day-to-day statistical error in QA happens, usually in the direction of declaring a difference that a second run would not reproduce.
- The hypothesis is stated before the data is examined, since a hypothesis chosen afterwards will always be supported by it.
- A p-value is read as the probability of the data given no effect, not as the probability that there is no effect - the reversal is the single most common misreading.
- Intervals are preferred to bare point estimates, because they carry the precision of the estimate rather than implying certainty.
- A 95% interval is understood as a property of the procedure over repeated sampling, not as a 95% chance the true value sits in this particular interval.
- Repeatedly re-running until a build passes is recognised as p-hacking by another name; the run that finally passes is a selection, not a result.
- Testing many metrics at once inflates false positives, so a correction or a pre-declared primary metric is used.
Quality Metrics & Reliability
The standard QA numbers, and what each one can and cannot support. Every metric here becomes misleading the moment it is used as a target rather than a signal.
- Defect density is normalised by size (per KLOC, per story point) so modules of different sizes can be compared at all.
- Defect Removal Efficiency - defects found pre-release over total found - is the honest measure of whether testing is working, and it can only be computed after the field data arrives.
- MTBF and MTTR are tracked separately: they answer how often it breaks and how fast you recover, and improving either has different costs.
- Availability targets are converted into an allowed downtime budget, which turns '99.9%' into roughly 43 minutes a month and makes the trade-off concrete.
- Flake rate is measured per test rather than for the suite, so remediation is aimed at the handful of tests actually responsible.
- The suite's false-failure rate is known, because a suite that cries wolf often enough stops being read at all.
Statistics Tooling
Where the analysis actually gets done. The tool matters much less than whether the sampling was sound - a well-run notebook cannot rescue a biased sample.
- Python with pandas, SciPy and statsmodels covers essentially all routine QA analysis and keeps the work in a language most test suites already use.
- R remains stronger for statistical modelling and publication-quality plots where the analysis is the deliverable.
- Optimizely, GrowthBook and Statsig handle assignment, sample-ratio checks and sequential-testing correction that hand-rolled A/B analysis usually gets wrong.
- GrowthBook and Statsig can run against your own warehouse, which suits regulated data that cannot be shipped to a vendor.
- Grafana charts percentile latency directly from Prometheus histograms rather than from pre-averaged values, which is what keeps the tail honest.
- Power BI and Looker Studio suit stakeholder-facing quality reporting where the audience will not open a notebook.
- Excel and Google Sheets handle descriptive statistics, PERCENTILE and basic tests perfectly well for small one-off questions.
- Spreadsheet percentile functions are checked for interpolation method, because they do not all agree with the one the monitoring stack uses.