I recently delivered a keynote address at one of those chi-chi-la-la invitation-only conferences. Just before my speech, this guy, Jack, in the second row calls out, “I hope I learn something today.”
I must admit, before I slam him to high-heaven, that Jack’s a decent guy. He runs a privately-held company that’s worth a little over a billion that he built himself with virtually no seed money. He sold it to a VC firm in the late 90’s and bought it back two years later because he didn’t like what they were doing with it.
He’s been to over a dozen of my seminars and he’s FINALLY starting to think that this internet thing may stick. (No, I am not kidding.)This is a guy, who until a couple years ago, had a secretary print out his email three times a day so he could scrawl all over it and return it to her for typing up and sending.
“I mean last time I implemented one of your ideas it was a bomb. Did NOT work whatsoever.” Jack added with a smirk.
What is it with these people? I mean, bring a voodoo doll or something but not right before a speech. Good grief!
Just out of morbid curiosity and basically because I just couldn’t resist, I retorted, “What did you test?”
“Abandoned carts. The program was worthless.” He all but shouted.
Nice buddy, real nice. The flipping topic of my speech was “38 Sure-Fire Tips for Increasing Your Conversion” and Jack The Ripper was attempting to destroy my credibility.
So I started asking him questions… One after another… like an FBI interrogator.
What did you mail? When did you mail it? What was the subject line? Was it personalized? And so on and so forth. It took me eleven questions but I finally found a kink in his armor.
“How many did you mail?” I inquired, not really noticing that EVERYONE in the room was listening intently.
“Seven.” Jack exclaimed triumphantly!
“Seven what? Seven thousand? Seven million? ” I asked.
“No, just seven.”
I burst out laughing. I thought he was kidding. He wasn’t. He was dead-as-a-corpse serious. You could have heard a pin drop.
“How could you mail seven and think that you had any kind of statistical validity or that you’d get any results at all for that matter?” I shot back. “I’m not trying to be rude here (I really wasn’t) but are you smoking crack? I mean, you have a $350 average order and you only mailed SEVEN PEOPLE!
If you were lucky, four of them actually received the email which means 1-2 people might have clicked on it – MAX. How exactly did you expect to calculate a quarter to half an order? And just for my information, how long are you leaving your cookies open and at what point do you expire your carts?”
The host was getting anxious about starting on time, so I told Jack we’d finish the conversation after my speech and we did. I’ll spare you all the gory details, but the discussion got VERY heated before Jack got the point. In his defense, this happens with a lot of my clients.
Online testing is difficult.
I will be the first one to admit that. It’s not at all like offline testing. There are many more variables that you have to fit into the equation, the data can be suspect, all sorts of things can impact it, and so on and so forth.
Plus, people tend to do one of two things. They either look at data that is unreliable or inaccurate …. or they use data that is good but they read it incorrectly. Even worse, a tester may often have such a strong theory going into the whole thing that his judgment is clouded. (It took me a long time to become objective.)
8 Tips for Improving Your Online Testing
1. Know whether or not you have a large enough sample size to test. Jack sent just seven abandoned cart emails and then threw out the program because it “failed miserably.” The guy built a big business so it’s obvious that he’s not taking the little yellow bus to work, but he’s not alone in his poor testing decisions. I see literally hundreds of people make the exact same mistake (although mailing only seven was definitely the worst). If you’re going to test something, you need to have significance. A test that ends up with 32 inquiries on one versus 33 inquiries on the other should most likely be thrown out or retested.
2. Do not test anything within 10 days of a major holiday. That’s 10 days before and 10 days after. If you are testing something seasonal (i.e. Christmas, especially) and you simply MUST test, do as many things as you can to isolate everything except a solo variable.
3. If something major happens in the world, throw out the test results. They’re more than likely going to be garbage. If you are www.nbc.com and you are doing a big test of your store, don’t run the test during the night of TheBiggest Loserfinale and think it’s going to accurately reflect what’s going to happen the rest of the year because it won’t.
4. Make sure to separate your online-sourced users from your offline-sourced users. This will make a huge difference. You don’t necessarily need to have an equal percentage of each but you do need to segment them so you can track them separately. Same with e-mail folks — no e-mails versus e-mails must be a segment.
5. Make sure the TOD (time of day) and DOW (day of week) is the same. Many times people come up to me and say — “I did so-and-so test and this was the clear winner.” When I look at the results, I realize that one test was conducted on Monday-Wednesday and the other was Thursday-Sunday… Or perhaps one was conducted from 6 am to 6 pm and the other went from 6 pm to 6 am. That’s not a test, especially on the Internet where time makes such a big difference.
6. Considering testing the same variable more than once. From an online perspective, backtesting the control is almost always a must.
7. Make sure the test is an exact A/B split. You can’t test email creative and landing pages all at once and expect to come out with valid results. Separate your variables and test them one at a time.
8. Most important, watch your metrics carefully. One of my clients recently threw out a 3-month test because there “weren’t more orders at the end.” After closely examining the results, I realized that although there were fewer completed orders, there were over 70% more attempted orders with the test version than with the control. After a retest, we found that the new landing pages were getting a lot more people to ATC (adopt to cart) which, with a good email follow-up process was instrumental in increasing their overall revenue!
Well said, Amy! Point number 8 is a big one. Google Website Optimizer recommends the “conversion” page be the cart, not the order thank-you page. That’s not what we direct marketers usually think of as a conversion, but there are too many new variables introduced when you wait until the end of the checkout process (the variable of a sucky checkout process, for instance).
Clients have also heard so many big claims on white papers and so forth that if a test doesn’t yield a huge, double-digit lift they yawn and consider it a failure. But I’d say, if an easy change like a different headline, a different colored button produces even a 5% or 7% lift or whatever, it’s found money — love it, welcome it, learn from it and test bigger ideas next time
Hello Tom Funk — Have I ever mentioned how cool it is that you read and comment on here? I’m very appreciative. Thanks for pointing out the Google Optimizer thing — it is an excellent point. As you know, I love the Google products but I am definitely not crazy about how they calculate much of anything that has to do with checkout (or abandons.) As for small changes making a BIG difference — you are right about that and your company (Timberline Interactive) does a great job with them. What about a guest post here sometime?
Amy – thanks so much. Althugh you and I consult in different fields – its nice to know that we have the same kind of patient, understanding, good listeners for clients
Thanks so much for this – #1 is huge, so many people don’t understand that. Most people think testing is an obvious thing – whichever one performs better is it! But there’s more to it, and I think this post points out that you test and re-test, and get to WHY it performed differently. There’s more going on in the world than just your A/B test, and there is an art to testing!
Hi Brandi. You’re right #1 really is HUGE and very few folks seem to realize it and/or care. I spoke to someone yesterday that was upset because I told them their test results were garbage. She had recommended to her boss that they go forward with the new site as they had run the test for a month. The difference between the old site and the new site was THREE ORDERS and they ran the test M-T for Group A and T-S for Group B (the old site.) They are B2B and get virtually NO traffic on the weekends! Nothing like stacking the deck, eh? Anyway, thanks for writing.