Most SaaS & e-Commerce founders and marketing teams are adept at managing and driving sales through online channels such as PPC, Paid Social, Affiliates & Partners, and SEO/Organic search.  However, at a certain point, most think about offline media such as TV, Radio, OTT or other customer acquisition channels.  How can you tell if these channels will work for your business?

The Economics of Offline Media

Offline media is purchased on an impression basis.  Specifically a CPM basis, which is a cost per thousand impressions.  If you purchase 100,000 impressions at $20 CPM, the total cost will be $20 X 100 = $2,000.  The difficult part about testing offline media is figuring out if it drove any traffic, and more importantly any purchases to your site.

With offline media you need to purchase and run a certain number of impressions in order to see any measurable effect.  This is different from online ads because online ads can often be tracked on a click-by-click basis.  But with offline media, you need to see a change in your baseline activity to be able to tell if it is working. For this to happen you need to have enough impressions to drive significant traffic.

During my time at Grasshopper, I personally wasted at least $150K-$200K on offline media tests that were never going to show an impact.  I did not know this when I ran them, but in retrospect it is obvious.

In my experience testing offline media, it takes between 3,000 and 4,000 offline media impressions to generate a single website visitor.  So, if you want to run a $50K local radio test with a $15 CPM over a 6 week period you can quickly see the difficulty in measurement.  With a $15 CPM you will get about 3.3M impressions which will lead to about 1,000 website visitors.  Taken over 6 weeks this is about 25 visitors per day to your website.  This would be a difficult effect to see on all but the lowest traffic sites.    

The Key to Offline Media Testing

Offline media testing is expensive.  It is unlikely that you can generate measurable results for less than several hundred thousand dollars for radio or TV testing.  And with this level of spend, the test will need to be concentrated in both time and geography.  It's best to work backward to figure out what a test should look like.

Offline media is purchased in DMAs (geographic regions that roughly align to the old metro areas in Google Analytics).  Let’s say that you have the following weekly traffic in these metros:

As a rule of thumb you will likely notice a 50% increase in traffic.  So if, for example, you are going to run a test in Phoenix, you will want to purchase enough impressions to drive about 633 users in a week or about 2.2M impressions at a cost of about $33K per week.  For a typical media test you would run 3 weeks on, one week off, 3 weeks on and one week off for a total of 6 paid weeks and a test duration of 8 weeks.  If you pick three of these cities in the table above, the total cost of the test will be about $550K - $600K.

If your traffic is lower than the example your test can be less costly but there is a lower bound.  For each city, you will want to purchase at least as many impressions as the size of the market. The ratio of total impressions to the market size is referred to as Gross Rating Points and you will want to ensure that you have purchased around 100 Gross Rating points per week in each test market.  The total weekly GRPs are shown in our table above.

Once you have selected your test cities you can create a baseline for the test cities as well as the balance on the US geographies.  The baseline will contain the weekly users going back 12-16 weeks for the test regions as well as the rest of the US.  Once you start your test you can measure the change in the test region over the baseline and compare it to the change of the balance of the US over the baseline.  This way you can be more certain that any change you are seeing is due to the media run and not seasonal or time-based changes.

This same process can be used for purchases or conversions.  The only difference is that you would substitute purchase or conversion data for user sessions in the analysis.

Conclusion

Offline media can be a game changer for a SaaS or e-Commerce business's customer acquisition, but few companies test this channel in a way that gives them solid data on the channel’s effectiveness.  Many run small tests that are not sufficient to show a result and conclude that the channel does not work for them.

However, before running a media test, it is important to figure out if the economics will work for your business even if the media performs well as a channel.  I will discuss this in my next post.

Mike Morris is the founder of Kettle Hole Partners. When he is not trying to figure out mathematical models for marketing and customer acquisition he is probably riding his bike.