Why is statistics really important?
While researching for this post, we came across an interesting quote about statistics:
“Statistics is like a bikini. What it reveals is interesting, but what it hides is vital”
Come to think about it, in some sense statistics is key. It helps us make important decisions and also nudge us in the right direction. At the same time, it could be misleading and manipulative to make us believe something. In summary, statistics have a special persuasion appeal that few people can resist.
Statistics is nothing more than the art of interpreting data to make meaningful decisions. In this post context, statistics will be specific to A/B testing.
We’ll learn the following:
- Population vs sample
- Measurements (mean, variance, standard deviation)
- Confidence interval
- Statistical significance
Shall we begin?
Population vs sample?
You’ve probably heard this term being tossed around like white fajitas bread. The reality is, it’s a very useful concept to understand and can make or break your whole understanding of statistics - so pay attention.
So what a “population”?
According to our friends at CXL, it simply means it’s the whole group of subjects you want to study. For example, say you want to compare two coffee stores: Store A and Store B - who has the hotter coffee?
The population would be ALL the coffee served by Store A and Store B every since inception. Keyword: ALL. if that’s 500, or 5 million so be it. It has to be ALL the coffee ever served.
If you can’t imagine, that’s going to be a lot of data collection and gathering to do. In fact, in some cases nearly impossible - how would you measure how many cups were served in 1997 if there wasn’t an inventory tracking system installed at the stores?
So the key takeaway here is that populations refer to ALL of the subjects you’d like to study. That often either means really really expensive or really impossible.
So what’s the next best thing?
Enter samples. Yes, samples.
We are not talking about the samples of cheese that you get while you’re shopping in a mall but in this case, we’re talking about the concept of samples in statistics.
In statistics, samples refer to a “selected” draw of the population where you assume will represent the whole population.
For example, instead of collecting data for all coffee every served by store A and B (population), we sample the first 10 coffee served the beginning of every hour in Store A and Store B throughout the month of December. We then assume this data will represent the whole population - which in reality it wouldn’t but it’s a simplified assumption.
So why are samples important?
Simply because they are realistic and is practical. You can imagine collecting every 10th coffee temperate in both Store A and B in December VS collecting info on the temperature of every single cup served since inception.
So are samples perfect? - heck no. But are they useless - heck no.
Samples give us a glimpse into what we call the “true” nature of the population without even studying the population - talk about a shortcut or a hack.
Once you’ve sampled your population - what can you do with it?
Gathering data without analysis is a futile effort. It’s the analysis that brings out interesting observations for you to act on.
There are many ways to dissect data but they are centered around two key questions: centered and spread
- Where are the values centred?
- How are the values spread?
Where are values centered?
Basically you want to ask yourself if I plot all the data I got from my sample - is there a specific number or value that the data tend to center around? A
The way to measure this is: mean
There are many types of mean: average, mode, median
But they all essentially help answer where the numbers are centered around.
Why is it important to know where the values are centered?
It gives an estimate of what is the snapshot of the values being analyzed. For example, we know the mean of temperate of coffee served in store A is 80 celsius while Store B is 120 celsius.
We can now safely assume that store B serves hotter coffee than Store A. So people who want to purchase coffee from Store B should always carry a coffee cup holder to withstand the heat (maybe heavy duty)
This is one example of how knowing how the values are centered can help make decisions.
How are the values spread?
Enter variance - the measure of how spread out the values are from the mean.
Basically is further the values are spread out from the mean the higher the variance.
Variance helps us gauge a lot of things - consistency, range, and even outliers
How can I use variance to make decisions?
You can use variance to gauge how consistent Store A and Store B serve their coffee. If Store A has a big variance compared to Store B - that probably means they are not consistent in serving their coffee at 80 celsius.
So if you’re curious you can find out if there is a specific pattern to look for that is causing the variance - specific barista or a specific time or specific machine that is causing variance - see how you can use it to understand a lot of things?
What about confidence interval?
Well, it simply means how confident are you that the sampled data is representative of the population.
First things first, there is no such thing as a perfect statistical solution. Samples will never be as accurate as analyzing the whole population - accept it and embrace it. But if statistics are applied well, then the numbers represented by the population could in theory be close to the population itself. But there will never be an absolute overlap between samples and population.
That’s why we need the confidence interval concept - to clearly state how confident we are that the mean of the samples is approximate to the population.
A 90% confidence interval means that you’re 90% sure that the mean of the average coffee temperature in Store A lies between 80 - 90 celsius (from your sampled data).
What about statistical significance?
You sampled every 10th coffee from Store A in December to figure out the average temperature. If we asked you - how likely will you be able to reproduce the same set of figures you sampled in a different experiment? What would your answer be?
No - panic is not a choice. Here is where the concept of statistical significance comes into play. It tells us how likely is the figures you sampled occur due to chance vs replicable.
We measure statistical significance through something called the p-value.
A 0.05 p-value means all the numbers I sampled have a 5% probability of being due to chance. Meaning 95% of being reproducible in a different experiment.
A high p-value means you won’t be likely to reproduce the same experiment - meaning it can’t be verified by someone else. Which doesn’t really matter if all you’re testing is coffee cup temperature? But it can mean life or death if you’re testing the efficacy of a new drug to cure a specific disease.
So what next?
I like to get people to play around with numbers for fun. Take a bunch of figures and look at where is the center and spread? And how confident are you of the samples and how likely is it due to chance?
As you ask yourself these questions - you’ll be able to make better decisions and analyze data like a pro.