So I've been toiling away on a new app icon for my guitar tuner and I figured I'd better test it against the old one before I go any further with the overall design of my app.
For some time now I've been known to suggest that people AB test their app names and icons using admob. However, until now I've been quite the hypocrite, having never done this myself mostly out of a combination of laziness, cheapness, and not having any other icons to test.
So I started an admob campaign with icon_old and icon_new ads and the same tag line in each. I decided to start with only $10 worth of clicks and if I didn't have statistical significance i would slowly add more until I got it. After a few minutes my $10 had been used up and I had gone through about 125,000 impressions per ad with 0.43 CTR for the icon_new and 0.36 CTR for the icon_old. If you do the math this equates to a significance level of 99.74% which even the most cautious would call significant.
I've been wondering how much it costs to get statistical significance using admob AB tests and now I know a ballpark. In this case statistical significance costs less than $10.