A/B testing with Jekyll and Google Analytics

Published on — Filed under protip, meta

As I was reviewing my last post, I realized it was long. So long I suspected most people would just give up after the first paragraphs — or until they noticed the scroll bar.

Being an in-depth technical post, there's only so much you can cut before it starts feeling like there are pieces missing. So I went with a TL;DR. Rationale was that if compelling results were immediately shown, people would bother to stick around to read the "how" & "why".

This blog is built with Jekyll so first thing I thought was "there's surely some plugin for that". Google said no.

The A/B test

The objective of this A/B test was to randomly show a TL;DR chapter and track each page view on Google Analytics with information on whether the element was displayed or not and analyze the effect of this element on the average time on page metric.

I inserted this right below Jekyll's headers on the post, before the rest of the article's text:

<div id="tldr">
  <p>
    <strong>TL;DR:</strong> Custom serialization proposed
    here is ~2x faster than Cocoa Archive Framework.
    Results are <a href="#results">here</a>.
  </p>
  <hr>
</div>

I'm not a big fan of introducing html into markdown posts, but in this case there was no choice.

Setting up Jekyll

After a bit of trial and error and some failed experiments, the solution was to have each post include an extra page template variable on the headers, which would be picked up by the top level template:

---
layout: post
title: "Cocoa data serialization benchmark: Archive framework vs custom serialization"
categories: [ cocoade ]
abtesting: [ tldr, 1 ]
---

That custom abtesting variable will be picked up by Jekyll and become available under page.abtesting.

You'll notice it's an array — let me remind you this is a very simplistic approach — with the values tldr and 1. This is information that will be passed along to Google Analytics, but we'll get there in a moment.

Displaying or hiding the TL;DR element with JavaScript

Whether this div was show or not had to be decided when the page loaded at the client side — Jekyll generates static html resources.

So, on default.html which is my top level template for Jekyll, I added a snippet to determine whether the page being displayed includes an A/B test — that is, if the post has that custom abtesting template variable set:

<!DOCTYPE html>
<html>
<head>
  <title>{{ page.title }}</title>
  <!-- other scripts, css -->
  <script src="/js/biasedbit.js"></script>
{% if page.abtesting %}
  <script type="text/javascript">
var abTestingId = '{{ page.abtesting[0] }}';
var abTestingSlot = {{ page.abtesting[1] }};
var abTestingValue = Math.random() < 0.5;
  </script>
{% endif %}
</head>
<body>
  <!-- content -->
  <!-- google analytics script -->
</body>

If the page does include an A/B test, three variables will be generated:

To hide the element, I added a small snippet inside that biasedbit.js file:

$(document).ready(function() {
  // Test if we're doing a/b testing on tl;dr
  if ((abTestingId == 'tldr') && !abTestingValue) {
    $('#tldr').remove();
  }
}

This is the piece of code that will remove the element with id tldr from the page as soon as the document finishes loading.

Tracking page views on Google Analytics with custom variables

Up until now, we've taken care of randomly displaying or hiding an element with id tldr on the page, so all that's left is actually track this information so we can later view the results on GA. Using the 3 variables introduced above, we can add a small snippet before the call to _gaq.push(['_trackPageview']); — on the top level template, default.html:

<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'YOUR-TRACKING-ID']);
// If we're doing A/B testing, make sure we include the proper variables
if (abTestingId) {
    if (abTestingValue) {
        _gaq.push(['_setCustomVar', abTestingSlot, abTestingId, 'A', 3]);
    } else {
        _gaq.push(['_setCustomVar', abTestingSlot, abTestingId, 'B', 3]);
    }
}
_gaq.push(['_trackPageview']);
...
</script>
</body>

Read up on Google Analytics custom vars.

Which these four little changes I got A/B testing with tracking up and running.

Wrap up: querying the data

To view the data gathered with this method, I had to create two custom segments. You can do that by logging in to Google Analytics, head on to your site and click "Advanced Segments" and then "New Custom Segment". Create something like this:

Creating an advanced segment

Then do the same for the 'B' value.

Finally, open "Advanced Segments" again and and enable these two filters you just created. You will now see both types of traffic.

So, does the TL;DR thing work or not?

Nope. No difference:

Results

I'll be testing this on more posts but I suspect that it won't make a difference. TL;DR or not, when people really are interested in in-depth articles, they'll stick around.