Beware: That hair’s not as shiny as it seems

Beauty product advertisers love to make claims about their effectiveness. "Shinier hair in days". "Reduces wrinkles". "Lifts your skin".
So bold and confidently made are these assertions that you'd expect there to be unequivocal scientific evidence backing them up. But that's usually not the case. All that supports most of these claims is a small scale survey of product users. In fact, recently I saw an advert where the claim was supported by a survey of just 24 users. Yup, 24. This is highly questionable 'evidence' because of a core market research principle - the margin of error.
Let me explain.

Bear the margin of error in mind because it tells you just how reliable the data is".

There are four important words in the world of market research:

• The ‘population’ is the entire group of people that you’re interested in, e.g. your target market, your customers
• A ‘census’means that you’ve measured the views of every single person in the population
• A ‘survey’means that you’ve measured the views of some, but not all members of the population (you’ve taken a ‘sample’)

A census is great as it gives you a definitive answer to your question – because every single person of interest has inputted, you can have great confidence in the results.  However, conducting a census is usually impractical because the population may be large (thus consulting everyone in it time consuming and costly) and not everyone in it can actually be engaged.

That’s where surveys come in.  They’re practical to conduct and give an indication of what you’d be likely to find in the wider population.  But how good an indication of wider opinion does a survey give?  To estimate this, we use a statistical concept called the ‘margin of error’.

The margin of error tells you the extent to which a survey result is likely to match what you’d see in a census.  For example, imagine that 50% of people answer ‘yes’ to a survey question and the margin of error is +/-3%.  This means that the survey result is likely to differ from the result you’d see from the entire population by no more than +3% or -3% (i.e. a census would see a result between 47% and 53%).  The margin of error needs to be considered in conjunction with the ‘confidence level’ which indicates how often the margin of error (which is after all an estimate) will be accurate.  For example, a confidence level of 95% (a common standard used in research) means that the margin of error will hold true in 95 out of 100 cases.

Now, back to the survey of 24 beauty product users. This sample size gives a 20% margin of error, which means that if 69% or less agree with the product claim, it could actually be a minority of users who support the assertion.  That doesn’t make for very good advertising of course.

So when you’re next designing a survey or using the results to make decisions, bear the margin of error in mind because it tells you just how reliable the data will be.

You can use the calculators below to calculate the margin of error.  The first can be used when planning a survey to see what margin of error different sample sizes would bring.  The second can be used post-survey to see the precise margin of error for a particular result.

[df-pre-moe-calc]

[df-post-moe-calc]