Click anywhere to close

How many results are needed for an effective poll?

Posted: May 31, 2013

Tell us what you think

Knowing how many results are needed for an effective poll brings us back to college where we took Statistics 101.  The answer is... as many as we can get.

Opinion Bias

Imagine if we had to go to our boss and give a level of certainty against choosing choice A out of 4 different possibilities.  With more data backing up the choice we are able to be more and more confident that we are presenting the right choice. With a smaller sample size we expose ourselves to opinion bias.  This occurs when the sample size disproportionately represents our target audience.  

If our target audience is 50% male and 50% female, then polling 4 women and 16 males will give us opinion bias. When you run your surveys or polls try your hardest to get your sample size to match as closely to your target audience as possible   You may even want to include questions about the respondents demographic details so that you can use that later on to filter results.

Sample Size Fluctuation

Now that we understand how critical it for our sample size demographic to match our target audience we now need to be aware of how much influence a small change to a demographic within our sample size will have on the survey.  Lets assume your target demographic is 25% latino, 25% caucasian, 25% asian and 25% african american.  You can't run an effective survey with just 4 respondants - even though you might have 1 respondent from each demographic group and thus have a properly balanced sample size.  The reason for this is that each respondent has their own opinions regardless of targeted demographic.  This can greatly skew your results in a way that does not properly represent your target.

Generally we recommend a sample size great enough so that a misplaced individual cannot swing the overal sample size more then 1% (or up to 5% if your budget doesn't allow).  This means that if one of your target demographics is only 1% of the total, then you would need a minimum of 100 respondents from that demographic alone.  This would require a minimum of 10,000 total respondents (properly balanced by demographic) to get a fair and accurate picture. If you wanted 1% accuracy then you would need a minimum of 100 respondents for that demographic with a total of 10,000 respondents.

Targeting Difficulty

We already know how important it is to have your targeted demographic adequately represented.  Now the challenge is in ensuring that your sample size meets the right criteria.  You might find that you get plenty of willing respondents from one category but not from another. This difficulty increases depending on the total number of demographic groups we are targeting.  if we only have 2 demographic groups for example (male and female) then it is a lot easier to get the sample size to closely match both of them.  If we have 10 demographic groups we are targeting then it will make it a lot more difficult to ensure all are balanced.

To overcome this difficulty we always shoot higher then the minimum and filter results afterwards.  15% - 20% overage is usually adequate enough to give you plenty of respondents and yet still not break the bank trying to get your survey size.  If your sample size is smaller then 1,000 then use 20%.  If the sample size is 1,000 or larger use 15%.  If your sample size is already larger then 10,000 then you can even use a smaller percentage. We also tend to favor stripping out smaller or overbalanced demographics.  For this we use the 5/5 rule.  We don't like to include demographics that are smaller then 5% of our total demographic or demographics that are more then 5x larger then others.  Meaning if the demographics are 10%, 15%, 25% and 50%... there is too large of a gap between the 10% and the 50% demographics.  When this occurs we try to better organize our target audience to keep the groups better aligned with each other and balanced to help in our information gathering.

Demographic is Everything

Ok - So let's recap and decide how many respondents we need.  We know that our sample size needs to match our target audience to avoid opinion bias.  We know that the sample size needs to be large enough so that we don't have large sample size fluctuation.  We also know that we need roughly 15% more respondents in general to account for targeting difficulties. Let's assume our target audience is 42% male and 58% female. With only 2 demographic groups that are fairly close together we really don't need a large sample size.  In fact we need a minimum of 42 males and 58 females to respond in order to get a balanced survey.  However 1 person in the males group can swing the results by about 2.3% if only 42 are polled.  A minimum of 100 males ensures that one single respondent cannot through results more then 1%.  100 males and 138 females is balanced according to our demographic (238 respondents total).  Accounting for 20% overage on both groups we need approximately 286 total respondents.

Most surveys are not that simple though.  Let's assume that we have 14% blue, 39% red, 23% green, 10% purple, 14% orange.  In this example the sample size is a little more difficult to work with. We start off with 100 respondents equally balanced.  This gives us 10 respondents in the purple group.  However this is a very critical poll and a single respondent in that group could swing the results by 10% respectively.  We will have to increase this groups sample size by a factor of 10 to get a minimum of 100 respondents.  We are now at a total of 1,000 respondents for the entire group. Accounting for 15% overage we get a total of 1,150 respondents needed.    

Copyright © 2017 Dimbal Software - All Rights Reserved
Dashboard | Software Modules | Support | Blog | Article Index

Links | NPS | Polls | SEO Scans | Test Plan | Triage