A/B Testing That's Easy To Use

How To Do A/B Testing Easily

Optimizely Home

Recently it has become shocking easy to set up A/B tests on your own website. A number of companies such as Optimizely and Visual Website Optimizer have made experimenting with your site simple, accurate and fun. I’ve spent the last two months running trials with both services and have been thoroughly pleased with the results.

What is A/B Testing?

Basically A/B testing is running an experiment on your site to see what works best in engaging your visitors. Say you would like to know which tagline impacts your visitors. With A/B testing you can allow a certain percentage of your visitors to see one option while the rest see another. The software will measure your visitor’s interactions with your site so you can determine what’s the best option.

I’d had heard about A/B testing and what an impact it had made with Google and Amazon, but it wasn’t until I read an article in Wired that claims that in the 2008 Obama campaign “it was estimated that a full 4 million of the 13 million addresses in the campaign’s email list, and some $75 million in money raised, resulted from Siroker’s careful experiments [A/B Testing].” So the campaign raised an extra $75 million because they tested. WOW! After reading that I knew I had to start running these tests, but wasn’t sure where to start.

Visual Website Optimizer and Optimizely

While there are some WordPress plugins that provide this, I could tell by reading through their descriptions and instructions that they wouldn’t be the easiest or quickest to implement. So I began researching companies and decided to give Optimizely and Visual Website Optimizer each a try. Both offer a 30 day free trial. I started out with Visual Website Optimizer and discovered changes that would increase my page clicks by 2 to 25%. Everything from theme thumbnails to headlines to colors to adding security seals and customer testimonials was experimented with and quantified.

My hunches were not always right, which is part of the point. You want A/B testing to reveal what speaks to your visitors best. With Search Engine Optimization (SEO) you’re trying your best to speak Google’s language so you can get more visitors. With A/B testing you’re making sure that your doing your best to connect with your visitors once they’re on your site.

Using Optimizely

At the end of my free trials, I decided to go with Optimizely. I really like the interface and it’s a bit less expensive per month which can be helpful too. To show you how easy it is to use, I’ll walk you through an experiment I set up recently and show you how to create your own and what I learned.

Create Experiment

To get started, click the “Experiments” link at the top of the Optimizely site and choose “Create New Experiment. Enter the URL you’ll be working with as well as a name for your experiment. Your completed experiments can be archived so you can review the results later so make sure you give it a title that will remain useful.

Create Experiment

Make Changes

Next you’ll be taken to a frame with a live preview of your site. Here you can interact with just about anything on the page. You’ll notice a box that appears around elements when you hover over them.

Optimizely Editor

Clicking the box will bring up a contextual menu that lets you choose what to edit. Here I’ve selected the thumbnail for our Progress theme. Since it’s an image, it give me options that are related to images specifically. I can swap out the image for a different one (or even several if I want to try a few). I can hide the image, make it bigger and so on.

Contextual Menu

Variation OptionsGenerally you want each change to be its own “variation.” You can set the options for the variation by clicking in the top left. One thing you’ll almost certainly need from this is to change the name of the variation. That way when you’re looking at your report, you won’t need to check and see what “Variation #3” was, you’ll know by its name. For any experiment you’ll need at least one variation so you can run a meaningful test.

Set Up Goals

This is the part where you measure the effectiveness of your variations. There is button labeled “Set Up Goals” in the top right where you’ll want to click to get started. The most basic metric to measure is “Engagement” which basically records how many people clicked on that page. Any click equals a “conversion” in regards to engagement.

Create Goals

Beyond that you can choose from numerous options to see if a form is filled out, if a URL is visited and so on. Goals are saved from one experiment to the next so you can easily reuse common goals across the site. I have goal for visiting each theme’s page. That way I can see how changes affect each themes views.

Traffic Allocation

The last bit to set before you begin running your experiment is how much traffic each experiment will receive. You can set that by “Options” and choosing “Traffic Allocation.” Here you can specify the % of that page’s traffic that each variation receives. This is important because your pricing with all of these services is based on the number of views each experiment receives.

For $19 a month, Optimizely will give you up to 2,000 monthly visitors. Now you’re saying I have way more traffic than that so is going to cost a fortune. If you’re running an experiment and someone visits your site, but doesn’t encounter the experiment, they aren’t counted against your total. Optimizely defines a visitor this way:

How are “monthly visitors” counted?
“Monthly visitors” are counted as the number of visitors that use Optimizely, by being bucketed into at least one experiment on your website.

You’ll want to be wise with the number of visitors you send through so you don’t quickly run out of your experiments for the month. That said, useful changes to the high traffic parts of your site will give you the biggest boost.

All traffic is free during the free trial period, so I started out making changes to my home page where I knew I received the most views. To reach a result quickly, I split my variations equally similar to this:

Allocate Traffic

Once that’s set you can save your changes and begin the experiment. Then you wait. Results are tallied in real time, but you’ll need to reach a point of statistical relevancy before you know if what you’re changing is effective or not.


For my experiment here, I tested out hiding the headline on our themes listing page. Adding the line “Easily Create An Amazing Site For Under $50!” to our home page increased our click through rate there by 17%. Our second most common landing page is our themes list so I wanted to see what effect having the same line there would have.

Thus far it has increased the percentage of clicks by 2% on our themes page. That’s no where near as large a difference as the home page, but it’s still significant enough to warrant further testing. My next experiment is to try some alternative headlines there and see if one of those is more impactful on the second page than the first.

Your Turn

So that’s how you go about setting up an A/B test using Optimizely. It’s really quick and simple and can give you actual data to back up your design decisions. Plus with a free trial, you have nothing to loose giving it a try.

What have you always wanted to test on your site? Let us know in the comments.

Submit your comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Promo 1: Get our lifetime membership, all themes*, and updates forever for only 95 USD.
Promo 2: Get a free premium theme when you host with us.
+ +