Google Analytics Gets New Testing Functionality: Content Experiments

Today the Google Analytics team made an important announcement for marketers and analysts: Google Website Optimizer will be fully integrated into Google Analytics. Google will be discontinuing Website Optimizer as a stand alone product (by August) and the testing functionality will be available to Google Analytics users only. The testing tool will be found in the Content section under the header “Experiments”.

If you are an existing Google Website Optimizer, you will automatically be in the first round of whitelists, but Google is not manually whitelisting users after the public roll-out begins. Also, it is important to note that Google will not offer a migration option for Google Website Optimizer users, if you have tests running you can run them up to August and then you will have to re-create the tests on the new Content Experiments.

Content Experiments Advantages (over Website Optimizer)

1. Only the control script will be necessary to run tests, the script that redirects the traffic from the original page to the variations. The standard Google Analytics tracking code will be used to measure goals and variations, including existing goals.
2. Google Analytics advanced segments and page metrics will be available along with optimizing goal conversions based on your test pages. That means we will now be able to segment our results based on any advanced segment, which can help understand for which visitors the winning pages works best.
3. Improved statistical engine for analyzing experiments, which will help making decisions faster about the winning/loosing pages.
4. Tests results will not appear for at least 2 weeks. According to Google this is a mechanism to encourage statistical significance and avoid fluctuations in traffic patterns.
   Post Script: The Google Analytics team clarified that “While we won’t declare a winner for the first 2 weeks, we will be sharing test results from the very first days. It’s a small distinction, but an important one.”
5. Tests will automatically expire after 3 months to prevent leaving tests running if they are unlikely to have a statistically significant winner.
6. “Dynamic Traffic Allocation” functionality: traffic will be shifted away from low-performing variations, over to higher performing ones. This feature can’t be turned off. This is to prevent poor-performing variations from doing extensive damage)

It is interesting to analyze the new features and the ones being left behind for now. By removing (almost all) tag implementations, forcing users to test 2 weeks (to take care of statistics), and allocating traffic dynamically, Google is clearly focusing on the beginners, making it simple for companies starting their testing efforts.

Another interesting change is that tests will be limited to 3 months. For experienced testers, it is clear that a test that runs more than 3 months will suffer from several statistical problems, as the data will be very scarce. But this will also stop websites that use Website Optimizer as a cloaking tool, where your original page is optimized for search engines and receives a small percentage of traffic and the variations are showed to users.

Content Experiments Features Not Available Or Missing

1. Multivariate Testing will not be available anymore. Google will rebuild this functionality at some point in the future (read this testing framework for more on the differences between A/B and multivariate testing).
2. Ecommerce transactions as the goal will not be available
3. Maximum of 5 variations per test: this is a very big disadvantage and will limit tests significantly.
4. Maximum of 12 tests per profile at this time: another big disadvantage for companies doing many tests. Post Script: The Google Analytics team clarified that “the maximum is 12 active/draft tests per profile. People can create more than 12 tests per profile over time, just not run them all at once.”
5. Implement the control snippet code once and leave it on the page for future tests. As of now, each test will have its own code.

Again, we can notice that features that would be extremely important for advanced users have not been added to the Content Experiments. Limitations such as variations limits, test limits per profile, one code per test, and a lack of multivariate testing are extremely problematic for advanced users. Hopefully Google will lift some of those barriers quickly.

Content Experiments Help Articles

Below are the official help articles from Google. In addition, also check this step-by-step guide on how to create A/B Tests using Google Analytics Content Experiemnts.

Overview Topic

• The Benefits
• Requirements & Sign In
• The Experiments Interface
• Elements of an Experiment

Run an Experiment Topic

• Life of a Typical Experiment
• Prepare
• Configure & Modify
• Track Progress
• Stop
• Respond to Results
• Metric Calculations
• Page Does Not Appear Correctly

Notifications and Errors Topic

• Code-Checking Errors