As you can see, at lower sample sizes, the margin for error is very broad, but as you get close to 1,000, the margin of error narrows substantially. With higher sample sizes, the difference in margin of error for a confidence interval of 95% and a confidence interval of 99% is very small, despite the CI of 99% providing 5 times as much confidence as 95% (there is a 1 in 100 likelihood of a difference being due to chance vs 5 in 100).
So, what does this mean for your research projects?
Implications for research
Firstly, it is worth considering when you may need to have greater confidence in your research findings. Are there specific results where you would be uncomfortable with having a 5% chance of producing a “false positive”? In such cases, increase the CI you use. This can be done after completing fieldwork, but if the business requires a high level of confidence, it would be better to account for this at the design stage and commission a larger sample size.
Of course, there are other times when 95% might be too high. This is usually in situations when the sample is relatively small; think B2B projects where a sample is hard to come by or projects with a limited budget. Reducing the CI will produce more statistically significant findings, albeit they may need to be used more cautiously.
More broadly, it would benefit researchers to be more transparent about how we use confidence intervals and margins of error. There are some research results which are so crucial to a business that the margin of error should be explicitly stated (instead of a generic mention of the CI on the 2nd slide of a presentation).
Consider, for example, a volumetric study which predicts 1y volume sales for an unreleased product. A client might be delighted to see the forecast surpass their action standard for product launch, but if the prediction really has a margin for error of ±11% at 90% CI, the figures could and should be assessed (and expressed) more soberly.
As researchers, we can be guilty of using terms that are rooted in advanced statistics without fully understanding their implications. It is our responsibility to get to grips with these concepts, apply them appropriately in research, and present them in a way that users of our research can understand. By engaging with stakeholders about margins of error and statistical significance, we can build a better understanding of how research reflects real-world behaviour and inspire more confidence in insight-led decision-making.