Yeah, you read the title right. Hacking. #Jswipe. Many would say that this wasn’t possible, but this little hack has been accomplished. An anonymous member who we will call “Jared” has been able to essentially hack Jswipe while utilizing a fundamental marketing technique: split testing.
Jared, an astute, attractive young Jewish gentleman (many would also call a scholar) has a problem. Despite having a relatively vast network of Jewish singles that he met through parties and the like, he still hasn’t found Ms. Right (or Ms. Right Now for that matter). He downloads Jswipe, goes on a couple of dates, and he shortly realizes that he’s not getting any more messages for whatever reason. Hmm… he shrugs it off and chalks it up to a bad week.
Now, Jared goes to Vegas. When he lands, he is smitten by more than 100 messages. Mazel Tov!
Jared finds that because he is in a different city, geographically his sample size (explained below) is now entirely new. This is great, but unfortunately, he doesn’t plan on meeting his life partner while partying in Vegas. He goes back to the drawing board and comes up with a brilliant idea after a couple of drinks.
“What if I change my picture while I’m back in LA? I’ll test several pictures while keeping my other variables constant (Kosher/not Kosher etc.). That way, I can find the best picture to get the most messages.”
The #Jswipehack is now official. All Jared does is download a new version of Jswipe, make a new profile, and add a new picture. Boom. He does this for about ten different pictures each for a duration of about a week or so, guaranteeing that he has a sample size that is large enough to direct his decision making.
Finally, he finds the “winning” image. He evaluates this as the picture that gets the most messages and interaction on the app. This picture is nearly 35 percent better than others and get’s the highest response rate.
Read below for more juicy marketing details on how Jared was able to split test his way to the love of his life.
The Three Fundamentals of a Split Test:
Split testing or A/B testing isn’t another crappy buzzword that marketers use to validate their jobs. It’s real. It’s here to stay. The concept is simple enough, yet it can be over-complicated fairly quickly. Thus, to illustrate this example, we’ll use the #jswipehack story to color in the general concept.
#1) Control your variables. Get that s*** locked down.
When you split or A/B test anything, an incredibly important aspect of testing is to control your variables as much as possible. In the example above Jared was testing only ONE variable at a time, e.g. his profile hero shot. Let’s say he mixed things up and tested three or four different variables—the description, headline, even name—that would have confounded the test because the results would have been inconclusive.
So, when split testing, it’s important to keep all of the other variables constant while you isolate one and see the results.
#2) Statistically Significant Sample Size.
Woah, that’s a mouthful! Alliteration aside, this is a seriously important part of split testing and statistically related decision making. Let’s say you came up with this awesome ad copy, headlines, creatives, and overall badass ads. You flight the ads in the early AM and let them run for two hours. You see there has been less than 200 clicks for each ad, but you notice that one ad is crushing the click-through rate game and it already has a conversion.
What do you do? Optimize by turning off the low performers and adjusting the bid (assuming CPC) up or down 10 percent to 15 percent on the top performer because, well, it’s working? Wrong.
Use basic statistical significance calculators. Seriously. A good one: http://getdatadriven.com/ab-significance-test
#3) Don’t chase the rabbit
Make sure you acknowledge other variables at play despite a “controlled” split test. One common way you can look at this is the time of day. Chasing the rabbit from a time of day perspective or not taking into account other variables is a big no-no.
Almost always, your cost per click and cost per acquisition will be much lower in the AM than in the PM. There’s a lot of theories that surround this, yet primarily it’s an inventory issue. On average, there are more eyeballs online in the AM than in the PM. This is a huge overstatement, yet it’s a trend seen in media buying since the Classmates.com days.
Due to the increased supply, sometimes media buyers see a pattern and immediately act on it. The takeaway: Wait until you have a good sample size across a wide range of time and then act on it. Jared knew about this and he made sure that he had ample time to get a solid statistically valid sample
Don’t chase the rabbit. Stay diligent until you can get a good sample size.
Now that you understand some split testing fundamentals, who can spot the one pitfall here that Jared didn’t take into account. What is it?
Extra Notes – Jared didn’t take into account that he is getting a higher frequency of his photos with the same sample set. The same user would see him about ten times, the higher the frequency the more likely the user would engage as he’s a familiar face.