Thanks for visiting the original Drilling Down web site!  

The advice and discussion continue on the Marketing Productivity Blog and Twitter: @jimnovo

Read the book-
First 9 Chapters: pdf

Get the book at

Relationship Marketing, Customer Loyalty, and Retention book

Customers Speak Up on Book & Site

Workshops, Project Work: Retail Metrics & Reporting, High ROI
Customer Marketing

Fresh Customer
Marketing Articles

8 Customer
Promotion Tips


Customer Retention

Customer Loyalty

High ROI Customer
Marketing: 3 Key
Success Components

LifeTime Value and
True ROI of Ad Spend

Customer Profiling

Intro to Customer
Behavior Modeling

Customer Model:

Customer Model:

Customer Model:
Recent Repeaters

Customer Model:

Customer LifeCycles

LifeTime Value

Calculating ROI

Mapping Visitor

Measuring Retention
in Online Retailing

Measuring CRM ROI

CRM Analytics:
Micro vs. Macro

Pre-CRM Testing for
Marketing ROI

Behavior Profiling

See Customer
Behavior Maps

Favorite Drilling
Down Web Sites

Book Contents

Contact Jim Novo

Questions?  Call
(cell) 727.895.5454


Drilling Down

Turning Customer Data into Profits

 with a Spreadsheet

Site and Book topic:

Maximizing marketing ROI with customer behavior analysis


Learn Methods, Metrics
(site map)

[ Home ]    [ FAQ ]    [ Download   [ Contact / About / Privacy ]

This is the faster loading site for slow connections.  The "pretty" version of this page is here.

Control Groups in Small Populations
Drilling Down Newsletter #108 1/2010

Drilling Down - Turning Customer
Data into Profits with a Spreadsheet
Customer Valuation, Retention, Loyalty, Defection

Get the Drilling Down Book!

Prior Newsletters:

Last month's newsletter covering control groups resulted in a lot of additional questions on this absolutely critical measurement technique.  One note in particular covered a lot of ground but specifically, asked if / how you can use control groups when you don't have enough folks in the population to create statistically significant sample sizes.  I say yes, as long as you don't need prediction - and with small test populations, you often don't.  Right?  

Hoping the stats people don't come out and crucify me, let's do the Drillin'...

Questions from Fellow Drillers

Control Groups in Small Populations

Q:  Thank you for your recent article about Control Groups.  Our organization launched an online distance learning program this past August, and I've just completed some student behavior analysis for this past semester.

Using weekly RF-Scores based on how Recently and Frequently they've logged in to courses within the previous three weeks, I'm able to assess their "Risk Level"-- how likely they are to stop using the program.  We had a percentage who discontinued the program, but in retrospect, their login behavior and changes in their login behavior gave strong indication they were having trouble before they completely stopped using it.

A:  Fantastic!  I have spoken with numerous online educators about this application of Recency - Frequency modeling, as well online research subscriptions, a similar behavioral model.  All reported great results predicting student / subscriber defection rates.

Q:  I'm preparing to propose a program for the upcoming semester where we contact students by email and / or phone when their login behavior gives indication that they're having trouble.  My hope is that by proactively contacting these students, we can resolve issues or provide assistance before things escalate to the point they defect completely.

A:  Absolutely, the yield (% students / revenue retained) on a project like this should be excellent.  Plus, you will end up learning a lot about "why", which will lead to better executions of the "potential dropout" program the more you test it and see results.

Q:  However, in light of your newsletter, I realized that we should probably have a control group with whom we do NOTHING (just as we did this past semester) in order to prove the effectiveness (or not) of the program.

A:  Correct.  Otherwise, you won't be able to make a valid claim to the "saved students". People can always argue a variety of other factors were in play - seasonality, topic, course sequence, etc.

Q:  Since the actual number of students is confidential, can you please tell me what percentage you would use for a control group if we had 400, 800, 1200, 1600, 2000, 3500, or 5000 students?  You mentioned 10% in your newsletter, but the population you were referring to exceeded millions.

A:  Well, there are online calculators you can use confidentially, there is an example right here.

If you don't understand the variables they are asking for, explanations at bottom of page, though this is very simple - what is confidence level and interval plus population size.

Q:  Our population is MUCH smaller, and each customer is therefore even more critical.  I don't want to recommend an unnecessarily large control group that would prevent us from retaining future students when we could see they were having trouble.  

I suspect that our defection rates will be lower 2nd semester than 1st since students should be beyond the "learning curve," so I don't think we can justly say that the program alone is the reason for lower defection rates if we don't use a control group.

A:  Yes, well, this desire to "get as much test as we can" was the main point discussed in the newsletter.  And that's the challenge with very small populations - to hit statistical confidence levels at say population = 500, you need over 300 or so in control.  Not so great.

So we go back to the question of company culture and how intuitively confident people will be with the results.  Do they in fact need true 
statistical significance for a program like this?

There is a way around the significance issue - repetition. The stats part of this is all about the "likelihood you get the same results again" - real important for drug testing, not so much for 500 folks in a marketing program.  The question you need to ask here is: do you really need "prediction"?  Or does prediction just make the whole test more complex and expensive than it's worth?  What if you repeated the test a couple of times and got roughly the same results, is that "proof"?

Here is what I might do.  I would ask whoever needs to believe in the results of this test a question like this:

"Let's say we took a random 20% sample of the students and excluded them from the marketing.  We apply the marketing to the other 80% and their retention rate is 15% higher than the 20% who had no marketing. We do this test 2 more times and the retention rate of students in the test is 13% and 17% higher than the students in the 20% who do not receive the marketing.  Would you at that point believe that without question, the marketing drives at least a 13% improvement in retention among students?"

Do you see where I'm headed with this?  The more times you repeat the test, the more confident you will be in the results - regardless of sample sizes and statistical mumbo jumbo. At some point, the reality of the differences between test and control performance has to be accepted.  It may help to define up front how many repetitions the "boss" needs.

Also, remember you have the same basic issue on an initial failure of the test.  You can't know with a small population how likely it is this initial failure is simply a statistical quirk, and you need to repeat at least 2X more and get 2 more failures before looking to abandon the effort as not viable.

One more tip, on this idea of sequencing / semesters / experience with the program.  There is no doubt in my mind that 2nd semester students would have what is called a "survivor bias" and be less likely to drop out; you will get the best performance in a program like this with 1st semester students.  So if at all possible, run the test / control on only 1st semester students , or segment by semester.

There is no doubt in my mind that 2nd semester students would have what is called a "survivor bias" and be less likely to drop out; you will get the best performance in a program like this with 1st semester students.  So if at all possible, run the test / control on only 1st semester students , or segment by semester.  But, just because you run it on only 1st semester students does not mean you don't have an effect in 2nd semester.  

Continue to follow test and control into 2nd, 3rd, 4th semesters and you may see the dropout rate of the original 1st semester group continue to widen versus control.  This is not only great for the profitability of the initial 1st semester program but also provides you the baseline you have to beat (control) for those 2nd, 3rd, 4th semesters when you decide to see if you can have an additional effect by intervening in those periods.

Hope that helps!


Have a question on Customer Valuation, Retention, Loyalty, or Defection?  Go ahead and send it to me here.

If you are a consultant, agency, or software developer with clients needing action-oriented customer intelligence or High ROI Customer
Marketing program designs, click here

That's it for this month's edition of the Drilling Down newsletter.  If you like the newsletter, please forward it to a friend!  Subscription instructions are top and bottom of this page.

Any comments on the newsletter (it's too long, too short, topic suggestions, etc.) please send them right along to me, along with any other questions on customer Valuation, Retention, Loyalty, and Defection here.

'Til next time, keep Drilling Down!

- Jim Novo

Copyright 2010, The Drilling Down Project by Jim Novo.  All rights reserved.  You are free to use material from this newsletter in whole or in part as long as you include complete credits, including live web site link and e-mail link.  Please tell me where the material will appear. 

What would you like to do now?

Get the book with Free customer scoring software at:     Barnes &

Find Out Specifically What is in the Book

Learn Customer Marketing Models and Metrics (site article list)


Note: This is the "slow connection" version of the home site.  It's intentionally sparse design allows faster page downloads.  To access this content in what some say is a more visually appealing format, click here.

Questions about any of the concepts on this site?  Call (cell) 727.895.5454.  Or e-mail me.

What Will I Learn
in the Book?

About the Author
Newsletter Sign-Up

Example of the
Drilling Down Method

See Drilling Down
Results in Action

Relationship Marketing
Customer Retention

Customer Loyalty

Get the Book
with Free Software!

Fresh Drilling Down
Related Articles

Advanced Customer
Modeling Articles

       [ Home ]    [ FAQ ]    [ Download ]    [ About / Contact / Privacy ]  

Welcome to the original Drilling Down web site;
recent advice and discussion are on the Marketing Productivity Blog and Twitter.

Contact me (Jim Novo) for questions or problems regarding this web site.   
Copyright © 2000-2021 The Drilling Down Project. All rights reserved.  Privacy Policy.