In our previous post about geo-targeted testing, we reviewed a test we ran using Facebook and Instagram (FB/IG) spend and measured the impact this spend had on driving sales to our Direct to Consumer (DTC) and Amazon sites.  As you may remember, there was no significant impact.  In this post, we explore a test of Amazon DSP ads that we ran within the Amazon platform to drive sales on our Amazon store.

The Problem

A $10M per year Amazon site could not scale ad spend on Amazon.  Any increase led to a significantly reduced Return on Ad Spend (ROAS).  It wasn't clear what drove sales on the Amazon platform.  As we showed with the FB example, Geo-Targeted testing could provide a way to measure the effect of DSP advertising.

The Test

In this test, we identified 5 metros and increased the DSP spend there by more than 10X.  The test ran for 8 weeks.  Our hypothesis was that the increased spend would drive incremental sales.  This was measured based on the shipping ZIP code in Amazon.  Spoiler alert: there was no increase in sales.

The results

Shockingly, there was no significant increase in sales as a result of the increased spend.  This was much more surprising than the FB test from our previous post because the ad platform is a part of Amazon's platform and is used by many large brands.  Below is a graph of the website sales in the targeted geography along with the percentage of DSP spend in the targeted areas.

There are quite a few things going on in this chart so I will walk through each of them:  

The blue line shows the percentage of Amazon sales in the geo where we tested the increased spend.  Sales are relatively stable in the test geography at about 16.5% both prior to any DSP spend and while the percentage of DSP spend was normally distributed across the geographies.

The orange line shows the percentage of sales in the test geo after the increase in DSP spend - you can see that sales in the test area actually decreased a little. Although this is very odd, it is within the normal variation that we see, so it is essentially remaining steady.

The gray line shows what percentage of sales we would expect in the test geo based on the increased spend (yellow line) and a ROAS of 2 on the increased spend.

The gap between the gray and orange lines shows that the increased DSP spend didn't have any impact on sales.

Conclusions

DSP ads do not appear to be an effective way to scale Amazon sales - we were spending about $4K per week in total when we started the test and about $400 per week in the test area.  During the test we increased the spend in the test geo to more than $4,000 per week and simultaneously decreased the spend in the rest of the geographies and still didn't see any impact.

One theory that explains this result is that Amazon is not a discovery platform for the product and category that was sold by this company.  In other words, people go to Amazon to buy this and similar products specifically and search for those products - like Duracell batteries or 3M Duct Tape - as opposed to categories such as batteries and tape.  If this is true, it would explain why increasing ads did not drive increased sales.  While Amazon may be a discovery platform for some products and categories, it may not be for others.  Make sure your marketing approach matches your category.

We have also identified other advertising platforms that do drive sales on Amazon.  These are member rebate program such as Fetch Rewards and others.  

What to do from here

All advertising mediums have a saturation point - a point at which additional spend is not longer cost effective for customer acquisition.  In order to scale your sales you need to understand where your spend is relative to this saturation point with each advertising channel (DSP, PPC, FB/IG, TV, Radio, etc).  You can find these points by running spend tests such as:

• Double your spend and measure the effects
• Stop your spend and measure the effects
• Increase your spend in a particular area and measure the results

To move forward, figure out what you want to test, develop your test plan and execute.  

Mike Morris is the founder of Kettle Hole Partners. When he is not trying to figure out mathematical models for marketing and customer acquisition he is probably riding his bike.