I recently came across a question regarding the validity of using stratified sampling in small populations. What if you have two targets you want to learn about. What if these targets are fairly small (such as in B2B businesses?)
Let’s say you are in the wholesale television business. You sell to two different kinds of small retail stores: Mom-and-Pop stores and Online retail stores. You want to learn about the wants and needs of these two groups. You have a list of the 100 top stores of these types in America and want to target this list. Online stores rule the list with a small sprinkling of Mom-and-Pop stores (end of an era, I know.)
You can do what a lot of people instantly think of doing—“I want to know the opinion of these two groups—so let’s poll 25 of each. Come to think of it, let’s poll the TOP 25 of each group. Then I’ll know what to do regarding my list of stores.”
Think again. If we asked the opinion of the top 25 Online retail stores and the top 25 Mom-and-Pop stores and got 100% participation (haha, for argument’s sake)– we would know the opinion of the top 25 Mom-and-Pop stores and the opinion of the top 25 Online Retail stores- but we would not know the opinion of the population of your list. Why? (Hint: has to do with sampling error.) Unless your two populations are evenly distributed, you would have to go down to #80 (8MM revenue) on the list to get 25 Mom-and-Pop stores to sample and down to to #29 (50MM revenue) on the list to get 25 Online retail stores. With such varied revenue, your two segment samples may behave radically differently from the mean of those in the greater population. Some information is fun to know—but if you are planning to make critical business decisions on this data, you are taking a gamble.
If you want to know the opinion of the population on your list as well as the opinion of your segments you would have to use random sampling or stratified sampling. Random sampling (randomly selecting a sample of your total population) would represent your population as a whole well IF you can get a large enough sample that you can be sure that your two segments will be represented fairly. Say you choose 50 retail stores out of a hat—what guarantees to you have that you won’t randomly select 48 Online stores and 2 Mom-and-Pop stores? With small population sizes and small sample sizes- the sort of population sizes B2B’s get a lot—such sample variances could happen. Would you be comfortable with making decisions on skewed data? Stratified sampling is the standard when you want to ensure your survey results to reflect the diversity (i.e. segments) in it. It’s great for B2B businesses that want to make the most of their research dollars and ensure that they get information on (1) the target population at large and (2) the segments in it.
Back to the scenario…
I look at my list and add up the Online stores: 72. Mom-and-Pops: 28. I need to learn about each of these groups and then put the information together to learn more about my population. To get a 5% margin of error and 90% confidence interval, I need 58 Online stores from my list to answer my survey. I will need 26 Mom-and-Pops to answer my survey (smaller populations require sample sizes in high proportion. This is the ideal sample and you need to manage your need for accuracy with the costs of doing the research. Most people don’t exceed 70 percent response rates, regardless of incentives they offer their survey-takers.)
Ok, so you have a sample size. You do your survey and learn a lot about Mom-and-Pops and Online retail stores. “Wow- their needs are really different but I would like to know what to do with this information when I want to make a decision for the entire population of my target market—my list of 100?” Here is where you can use weighted averages to reflect the population. Say, for Mom-and-Pops, the mean level of interest in new packaging is 8 out of 10. For Online retail it’s 2 out of 10. Should I change my packaging? Your top-25 lists mentioned at the top of this article would make this idea seem like a good idea. But look at the weighted average of interest, taking into account the huge proportion of Online stores over Mom-and-Pops…
Number of Online stores: 72. Number of Mom-and-Pops: 28
Interest in new packaging for population=.72*2+.28*8=1.44+2.24=3.68
While as a good marketer, you need to balance customer wants with your budget constraints, brand building, targeting, positioning, etc, an interest level of 3.68 out of 10 is a solid data point to keep in mind when prioritizing your marketing plans. Unlike random sampling, because you collected statistically significant samples of the two segments, you know where the 3.68 came from and how to think about it. As you know, building decisions on solid data is critical- and stratified sampling gives you the solid data you need to make good decisions.