InsightIQ Blog

Control Yourself! Experimental Design for Marketing

View more Blog Posts

Aug 20 2010

First let me apologize for the sensational title of this posting. I am fully aware that this title does nothing to move us as a culture away from the hyper-emotionalism that has driven marketing for quite some time now. Then again, since marketing does strive to captivate the heart as well as the mind, I feel somewhat justified. I'll get right to the point. There are two reasons why this blog is worth your time:

1) You are a CFO with a company meeting rapidly approaching and you are wondering if all the dollars you are throwing at the Marketing department are yielding any fruit, or...

2) You are on the other side of the equation and you are a Marketing Director having to bear the burden of obtaining the elusive ROI measurement for the CFO. 

If you can relate to being in the first group, then file this posting under "knowledge is power". If you are in the second group then consider this posting...well, just consider it so you can avoid the following scenario. If you are a CFO it is likely that you are cringing because your next meeting with the Marketing Director is coming up and it will likely go much the same way they have for the past year. The Director will walk you through a deck that includes lots of line graphs and bar charts of different colors that seem to resemble the flight pattern of a fruit fly rather than the gradual black line heading north you hoped to see. When you inquire, you are sent to the second to last before the "Questions?" slide entitled "Root Causes" or something similar which includes a bullet that blames the poor economic conditions. Visions of the next SEC earnings call, the stock price, etc. all begin to fuel your anxiety; but before you draw conclusions about the strategy and execution of the marketing plan, consider the way you are measuring the data that draws those charts.

One effective method of measuring just about anything, including campaign results, is called Experimental Design. In a nutshell, it is a method of reporting that basically holds a population of your customer base under a microscope (called a 'study group') and measures the reaction of those who were marketed to (called the 'test group') vs. those who were not (called the 'control group'). The 'lift' is simply the percentage difference between those who responded and those who were not marketed to. The basic question it attempts to answer is, "Did our marketing efforts have any impact at all on consumer behavior?" Keep in mind that results that show attrition are just as important to recognize as results that show retention or cross-sell. The concept is simple enough to understand, and many companies utilize this method as their reporting standard. From here on out I'll assume you already use it. If anything I have said to this point is new to you feel free to contact me and I'd be happy to discuss.

To determine where the fruit fly in the ointment may lie, you must revisit the decisions made to determine the study group population, control group, pre/post measurement periods, types or segments of customers/prospects to use, effects of mediums that cannot be measured in a tangible way, etc.

This will take some work to unravel. If re-evaluating your reporting seems unnecessary because you have a firm handle on why a trend cannot be formed, then please stop here. For the rest, let's take a slightly deeper dive into the control group.

One question to ask is, "What populations should be in the study group?" It's a given that the group that received the marketing stimulus or 'test' group should be included within the population being studied. The more complex question is, "What population should be in the control group?" The most effective method I have seen here is to implement a static control group. A static control group is a designated group of customers that will not receive marketing stimulus for a specified amount of time. The object of a control group is to keep one group as pure as possible. The more touch points the control group has, the more opportunity it has to be more like the test group and therefore blur the results. This is why many marketers have chosen a static group- it is easier to manage marketing campaigns from different departments with different targets if one common group is used. This method is also the best way to keep the control group pure. The amount of time a customer remains in the control group is best determined by your comfort level around how long you feel a customer can go without hearing from you before they will begin to wonder if you value them. This depends on your type of business and the service or product offered by your company. There are no rules set in stone to follow here because no one knows your company better than you. One way to view this is for more expensive, high commitment services or products that require long term contracts like hosting a technological solution, you may choose a six or twelve month rotation since a customer is locked for a certain amount of time and it is likely that you have a daily working relationship with this customer. If you sell services or products that can be obtained easily from multiple sources, and do not cost too much - hence the lesser the commitment - then you may want to choose a lesser rotation since you need to keep your brand in front of the customers more often. You will also need to adopt a policy that determines the amount of time a customer should stay out of the control group after being in it. See this as a tour of duty scenario. Soldiers are sent out and then return home for a time to recharge before being sent back into battle. Customers will spend time outside the control group so they can be stimulated again before returning.

The diagram below shows how the groups above are divided up to create the populations for your reports:

Study Group Diagram

This is a good stopping point. Thus ends the first of an unspecified amount of installments. This series of blogs will resemble our great country- they will be built democratically. If I get enough responses (votes) I'll keep it going. I'll do my part to turn out the vote for the next election cycle by giving you some practice here. So do your civic duty and comment America! I'd enjoy your feedback. 

This article is filed under:

View more Blog Posts

Leave Your Comment Comments

Aug 23 2010

MJ said:

You're right on target. You have to use control groups to measure effectiveness. Unfortunately that's not done enough in actual practice.

Leave a Comment

  • Your email won't be published on our site.

All Fields Required