A/B testing with Jekyll and Google Analytics
Published on — Filed under protip, meta
As I was reviewing my last post, I realized it was long. So long I suspected most people would just give up after the first paragraphs — or until they noticed the scroll bar.
Being an in-depth technical post, there's only so much you can cut before it starts feeling like there are pieces missing. So I went with a TL;DR. Rationale was that if compelling results were immediately shown, people would bother to stick around to read the "how" & "why".
This blog is built with Jekyll so first thing I thought was "there's surely some plugin for that". Google said no.
The A/B test
The objective of this A/B test was to randomly show a TL;DR chapter and track each page view on Google Analytics with information on whether the element was displayed or not and analyze the effect of this element on the average time on page metric.
I inserted this right below Jekyll's headers on the post, before the rest of the article's text:
<div id="tldr">
<p>
<strong>TL;DR:</strong> Custom serialization proposed
here is ~2x faster than Cocoa Archive Framework.
Results are <a href="#results">here</a>.
</p>
<hr>
</div>
I'm not a big fan of introducing html into markdown posts, but in this case there was no choice.
Setting up Jekyll
After a bit of trial and error and some failed experiments, the solution was to have each post include an extra page template variable on the headers, which would be picked up by the top level template:
---
layout: post
title: "Cocoa data serialization benchmark: Archive framework vs custom serialization"
categories: [ cocoade ]
abtesting: [ tldr, 1 ]
---
That custom abtesting
variable will be picked up by Jekyll and become available under page.abtesting
.
You'll notice it's an array — let me remind you this is a very simplistic approach — with the values tldr
and 1
. This is information that will be passed along to Google Analytics, but we'll get there in a moment.
Displaying or hiding the TL;DR element with JavaScript
Whether this div was show or not had to be decided when the page loaded at the client side — Jekyll generates static html resources.
So, on default.html
which is my top level template for Jekyll, I added a snippet to determine whether the page being displayed includes an A/B test — that is, if the post has that custom abtesting
template variable set:
<!DOCTYPE html>
<html>
<head>
<title>{{ page.title }}</title>
<!-- other scripts, css -->
<script src="/js/biasedbit.js"></script>
{% if page.abtesting %}
<script type="text/javascript">
var abTestingId = '{{ page.abtesting[0] }}';
var abTestingSlot = {{ page.abtesting[1] }};
var abTestingValue = Math.random() < 0.5;
</script>
{% endif %}
</head>
<body>
<!-- content -->
<!-- google analytics script -->
</body>
If the page does include an A/B test, three variables will be generated:
abTestingId
: the identifier of the A/B test;abTestingSlot
: the slot for the custom variable, required for Google Analytics;abTestingValue
: a randomly calculated boolean, that decides whether version A or B of the site will be shown.
To hide the element, I added a small snippet inside that biasedbit.js
file:
$(document).ready(function() {
// Test if we're doing a/b testing on tl;dr
if ((abTestingId == 'tldr') && !abTestingValue) {
$('#tldr').remove();
}
}
This is the piece of code that will remove the element with id tldr
from the page as soon as the document finishes loading.
Tracking page views on Google Analytics with custom variables
Up until now, we've taken care of randomly displaying or hiding an element with id tldr
on the page, so all that's left is actually track this information so we can later view the results on GA. Using the 3 variables introduced above, we can add a small snippet before the call to _gaq.push(['_trackPageview']);
— on the top level template, default.html
:
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'YOUR-TRACKING-ID']);
// If we're doing A/B testing, make sure we include the proper variables
if (abTestingId) {
if (abTestingValue) {
_gaq.push(['_setCustomVar', abTestingSlot, abTestingId, 'A', 3]);
} else {
_gaq.push(['_setCustomVar', abTestingSlot, abTestingId, 'B', 3]);
}
}
_gaq.push(['_trackPageview']);
...
</script>
</body>
Read up on Google Analytics custom vars.
Which these four little changes I got A/B testing with tracking up and running.
Wrap up: querying the data
To view the data gathered with this method, I had to create two custom segments. You can do that by logging in to Google Analytics, head on to your site and click "Advanced Segments" and then "New Custom Segment". Create something like this:
Then do the same for the 'B' value.
Finally, open "Advanced Segments" again and and enable these two filters you just created. You will now see both types of traffic.
So, does the TL;DR thing work or not?
Nope. No difference:
I'll be testing this on more posts but I suspect that it won't make a difference. TL;DR or not, when people really are interested in in-depth articles, they'll stick around.