Skip to main content

What To Do When Conversion Optimization Tools Fail

By September 26, 2012May 12th, 2014Conversion Rate Optimization

I’m a big proponent of adaptive web design and conversion rate optimization.  I love the tools that are out there such as Google’s ex Opmitizer, Visual Website Optimizer and Unbounce. But one thing I’ve learned over time using these tools is that they are subject to making inaccurate claims about a winner, giving the illusion that one concept is better than another.

Cow on a Fence

Beware of Your Conversion Results

The primary issue I’ve found is that these tools are of course just software that follow rules that a programmer has given them. They can be fooled into thinking there is a winner if there is a lot of traffic and one of the designs is converting a lot faster than the other design.  In many cases, the tool you use is probably right.  But I’ve found that sometimes, if you wait long enough, those early winners are not really winners.

What happens, especially on sites that don’t change much, that have a loyal fan base, introducing something new is like bugs to a zapper. It is very attractive because it catches their eye and they wonder what it is and click through. So early on it will easily win out in this scenario and the optimization tool will declare a winner. But if you keep testing longer, if visitors don’t really care about the change, you’ll see conversions drop back down to normal levels.  Its very possible that the new design will win, but not at the initial levels, but it is also possible that it is worse than previously.’

And If You Are Testing Your E-commerce …

Another similar problem arises of when you run a test, especially for e-commerce websites. Just because something converts better during one part of the year, does not mean it will necessarily convert better at another time of the year.  For instance, the use of different colors on the site (green and red during Christmas) may work great during the rush, but wouldn’t work well at other points in the year. Its about timing of what you test and recognizing that different things work different based on time of year and geographic region.

A Simple Fix

So, the result of this is that you shouldn’t stop the testing if you get an early winner. Keep testing longer so that it normalizes and you get out all the bugs. Then you may want to try the test based on time of year or geography to help drive up those sales.  A little bit of personalization can really impact the bottom line in adaptive web design.

John Paul Mains

John Paul Mains is the Chief Marketing Scientist at Click Laboratory. He loves all things digital, but especially SEO and analytics. If you're interested in learning more, his LinkedIn profile is

One Comment

  • Hi John,

    You are right you should know what the calculations are of a tool. We have default 97% significance and 10 goals minimum in But we allow you as expert to tweaks both numbers as well as minimum and maximum duration. But we do have to set a default for the 90% of the users that do not want to tweak these settings.

Leave a Reply

Share This
  • Get started

    This field is for validation purposes and should be left unchanged.

  • download-now
    Step by Step SEO Conversion Checklist

    • This field is for validation purposes and should be left unchanged.

    Easy to Print for Daily Use