After running several Full page Optimizely A/B tests we began to receive feedback from our users that experience was degraded. Our pages were flickering when the b experience was rendering above the fold. Client side testing with Optimzely works well for smaller changes or if the changes are out of the visitor’s viewport when the page loads. It starts to be less ideal though when there are large areas of content above the fold. Depending on your site it can take a second or two for those changes to appear. We needed a solution on the back end that could serve up two different pages to our end users with no degradation to the experience.
Using Fastly to split the traffic on the edge allows for a seamless experience for our end users. This technique can be applied on any modern CDN like Cloudflare or with an edge serverless platform.
Authoring A/B Tests in Drupal
Inside of our content management system Drupal we wrote a custom module to send A/B test metadata to Fastly’s API. This allows our authors to set up tests without any CDN configuration or engineering help. The module adds a A/B Test tab (menu local task) on our pages to manage tests.
To create a B version of a page, A user can clone the current page which can be modified for the test. Author’s can also create a new page from scratch to test against. A lot of times we use static HTML includes to create new markup outside of our component library to test with.
To enable a test the user is presented with a form that captures the test name, A and B urls and the split percentage. When the user clicks enable, Drupal submits the values to Fastly’s API to make the test active and purges all of the paths.
CDN Configuration
The CDN needs to know three pieces of information for each test.
- The A and B backend URLs
- The cookie name for persistent routing
- The percentage split between A and B
Most CDNs offer a persistent key/value store for information like this. We use three Fastly dictionaries to store the data all keyed with the request url path.
| Name | Key | Value |
| A/B Test Path | /a-page | /b-page |
| A/B Test Cookie | /a-page | ab_test_sample |
| A/B Test Split Percentage | /a-page | 50 |
The three dictionaries all use the A page’s path as the key.
The A/B Test Path value is the B page’s URL that is used to backend route to load content and store it as the B page’s cache key.
The A/B Test Cookie dictionary is used to store the test ID transformed to lower case and with underscores instead of spaces. This value is used to detect a cookie that is set from the Drupal module that lets us know if a user has previously visited the page and helps persist the A/B experience.
The A/B Test Split Percentage dictionary value has the percentage of traffic the B page should receive. The percentage value can be used to send 100% of traffic to the B page once a test has won and the new components and translated pages are being built. Another use case is to maintain 10% of traffic to an existing page when a new design is rolled out. This allows our optimization manager to measure conversion rates against a new page vs the old one with the same amount of seasonal traffic.
In order to maintain two cached versions of the same request url we add a http.X-AB-TEST header with an a or b value. This allows us to append X-AB-TEST to the http.vary header which allows the correct version of the page to be fetched from the cache.
Measuring Test Results
Once a test is enabled we include a AB-Testing-Experiment-Viewed event to both the A and B pages. Optimizely tests send this same event with a variation name value (Original/Variation #1). We can then use our Snowflake EDW or GA to measure which experience improved conversion.
Final Notes
We have been running Fastly A/B tests for just over a year now and have learned a ton. Early on we had several caching bugs when enabling tests. Due to the multiple layers of CDNs and caches we had to implement delays in the purges to be certain that both the A and B backend pages were correct. In some instances the B page would have stale alt-href language meta tags or missing analytics events. We solved these errors by setting up an end to end testing environment to make changes on.
The choice whether to run a test on Optimizely vs Drupal comes down to ease of implementation And end-user impact now. We have been really happy to have both options in place.