Marketing Research: Surveying Small Populations with Diverse Segments

I recently came across a question regarding the validity of using stratified sampling in small populations. What if you have two targets you want to learn about. What if these targets are fairly small (such as in B2B businesses?)

Let’s say you are in the wholesale television business.  You sell to two different kinds of small retail stores: Mom-and-Pop stores and Online retail stores.  You  want to learn about the wants and needs of these two groups.  You have a list of the 100 top stores of these types in America and want to target this list.   Online stores rule the list with a small sprinkling of Mom-and-Pop stores (end of an era, I know.)

You can do what a lot of people instantly think of doing—“I want to know the opinion of these two groups—so let’s poll 25 of each.  Come to think of it, let’s poll the TOP 25 of each group.  Then I’ll know what to do regarding my list of stores.”

Think again.  If we asked the opinion of the top 25 Online retail stores and the top 25 Mom-and-Pop stores and got 100% participation (haha, for argument’s sake)– we would know the opinion of the top 25 Mom-and-Pop stores and the opinion of the top 25 Online Retail stores- but we would not know the opinion of the population of your list.  Why? (Hint: has to do with sampling error.) Unless your two populations are evenly distributed, you would have to go down to #80 (8MM revenue) on the list to get 25 Mom-and-Pop stores to sample and down to to #29 (50MM revenue) on the list to get 25 Online retail stores.  With such varied revenue, your two segment samples may behave radically differently from the mean of those in the greater population.  Some information is fun to know—but if you are planning to make critical business decisions on this data, you are taking a gamble.

If you want to know the opinion of the population on your list as well as the opinion of your segments you would have to use random sampling or stratified sampling. Random sampling (randomly selecting a sample of your total population) would represent your population as a whole well IF you can get a large enough sample that you can be sure that your two segments will be represented fairly.   Say you choose 50 retail stores out of a hat—what guarantees to you have that you won’t randomly select 48 Online stores and 2 Mom-and-Pop stores?  With small population sizes and small sample sizes- the sort of population sizes B2B’s get a lot—such sample variances could happen.  Would you be comfortable with making decisions on skewed data?    Stratified sampling is the standard when you want to ensure your survey results to reflect the diversity (i.e. segments) in it.  It’s great for B2B businesses that want to make the most of their research dollars and ensure that they get information on (1) the target population at large and (2) the segments in it.

Back to the scenario…

I look at my list and add up the Online stores: 72.  Mom-and-Pops: 28.  I need to learn about each of these groups and then put the information together to learn more about my population.  To get a 5% margin of error and 90% confidence interval, I need 58 Online stores from my list to answer my survey.  I will need 26 Mom-and-Pops to answer my survey (smaller populations require sample sizes in high proportion.  This is the ideal sample and you need to manage your need for accuracy with the costs of doing the research.  Most people don’t exceed 70 percent response rates, regardless of incentives they offer their survey-takers.)

Ok, so you have a sample size.  You do your survey and learn a lot about Mom-and-Pops and Online retail stores.  “Wow- their needs are really different but I would like to know what to do with this information when I want to make a decision for the entire population of my target market—my list of 100?”  Here is where you can use weighted averages to reflect the population.   Say, for Mom-and-Pops, the mean level of interest in new packaging is 8 out of 10.  For Online retail it’s 2 out of 10.  Should I change my packaging?  Your top-25 lists mentioned at the top of this article would make this idea seem like a good idea.  But look at the weighted average of interest, taking into account the huge proportion of Online stores over Mom-and-Pops…

Number of Online stores: 72.  Number of Mom-and-Pops: 28

Interest in new packaging for population=.72*2+.28*8=1.44+2.24=3.68

While as a good marketer, you need to balance customer wants with your budget constraints, brand building, targeting, positioning, etc, an interest level of 3.68 out of 10 is a solid data point to keep in mind when prioritizing your marketing plans.  Unlike random sampling, because you collected statistically significant samples of the two segments, you know where the 3.68 came from and how to think about it. As you know, building decisions on solid data is critical- and stratified sampling gives you the solid data you need to make good decisions.

Cyworld To Shut Down Its US Service

I guess I saw this one coming.  On November 5th, I received the following email notice:

Thank you to all members with Cyworld.

Due to Cyworld shuts down US service, US Cyworld will no longer be able to service.
We sincerely apologize for shutting down the service with unavoidable reason.
Before US cyworld close the service, you will continue to access to US cyworld contents but not
purchase items. Also, you will not use your acorns.
If you have unused acorns, you will be given a full refund for paid acorns only.

Refunds and data backup service is in progress, using the acorn will no longer be able to purchase for miniroom items, skins, etc.

@ Schedule for closing US Cyworld service
Due to Data Back-up and closing service issues, the service will be unavailable.

* Shop service will be unavailable since Nov 03, 2009
o Club service, Profile photo/data upload serivce will be unavailable since Nov 23, 2009

While I loved the oh-so-cute  “minime” (avatar) and “miniroom” (avatar’s house) that users could design for their pages (I couldn’t help but share mine, below)– the only reason why I signed up for Cyworld was so that I could connect with my Korean friends (plus I was curious.) Unfortunately, because Cyworld actually chose to separate these two networks geographically (can you imagine facebook doing this?)- it eliminated the whole point of using the site.

I found out today from a fellow blogger that you can sign up for the Korean Cyworld site– and (get this) they have recently eliminated the requirement that you send in a copy of your passport for approval before you get to sign on…

My takeaway:  Network effects in social media are not to be ignored.  Unless you have something compelling to offer, are focusing on a market niche or are first-to-market, don’t assume you can start a platform at zero and take on an already saturated market.  Use all the network connections you can.  Think: would you build a new line of fax machines using machine language incapable of working with any other models– including your own? No way.

Your thoughts?