Skip to main content

A / B Testing

To improve user experience, Atomic enables A/B testing of cards, and then the ability to analyze test results quickly and easily.

How you achieve this will depend on your preference and resources available to you. There are three suggested patterns:

1. Events filter​

Creating an events filter is the recommended pattern, as it makes the analytics easy to compare between card versions.

It requires your engineers to build a filter for your events before sending them to Atomic. The filter would randomly choose a version of the card to trigger, and send the corresponding event to Atomic.

Set up an events filter

  1. Ask your engineers to build a filter to randomly segment your trigger events, before they are sent to Atomic.

  2. In the Workbench, author a card template (A). See: Author a card template, including a card trigger.

  3. Duplicate the card template to author the B version:

    • copy the ‘Raw card data’ (in the card content editor, under ‘advanced’)
    • author a new card by going to ‘Cards’ and then ‘New card’
    • paste the ‘Raw card data’ into the new card.
    • don’t forget to publish the card. See more on publishing in managing cards in Atomic

  4. Edit the B version to match your test requirements. This could be any compilation of layout, media, action buttons, or copy etc.

  5. Add a separate trigger and event to the B version, ensuring it is sent to the same stream.

  6. Publish the cards, and then have your systems send the branched versions of the event to a different trigger: for the A or B versions.

2. Manual switch​

Rather than running A/B tests concurrently, you can manually switch the card at a specific time, and then segment the analytics data using time filters.

The length of time to run each segment will be determined by your sample size and level of confidence desired in the results.

Set up a manual switch

  1. Pick a time period to run the test that minimizes the effect of other variables. E.g. control as best you can for day of the week, time of day, usage of your app, and any other significant events for your customers and company.

  2. In the Workbench, author a card template (A). See: Author a card template and a corresponding card trigger.

  3. Prior to the agreed switch over time, begin making changes to the card for the B version. Note: changes to the card will not be sent to customers until the card is re-published. See more on publishing in managing cards in Atomic.

  4. At the agreed switch over time, re-publish the card with the changes desired for the B version.

  5. Once the experiment is complete, segment the analytics data based on the switch over time/date.

3. Variable switch​

Card copy

This method is only suitable if you’re A/B testing based on differing card copy. For example, what is said in the headline, body, or action button. You can not use this method if you wish to test different versions of media, or the number of action buttons used on a card.

Atomic provides a mechanism to add metadata to events, as the detail of any variable in a card is included in analytics event payloads.

For example, if you add a variable called copyVariant to a card, then in downstream analysis, the version of the card would be identified and segmented by the value of the copyVariant variable, which will be included in all the related analytics event payloads.

Set up the variable switch

  1. Author a card template. For this method we use the same card template and trigger for both A and B versions. See: Author a card template and card triggers.

  2. Add a variable, for any copy you wish to test in your A/B test. E.g. the copy on a button, or within the headline. See: Add a variable

  3. When your engineers send events to Atomic, the payload contains a different version of the variable.

  4. Segment your analytics based on the variable used (returned in the analytics payload).

Analysing A/B Testing results​

For the events filter and manual switch methods, the - Insights area provides the quickest and easiest way to analyze your data. Here you can easily segment the data based on date, stream, or platform.

Rich analytics data to plug into your existing analytics platforms is also easily available through either batch download, or API. See more about getting analytics data from Atomic at: Analytics from Atomic.

First class feature

We’re currently thinking about ways to support A/B testing as a first class feature in Atomic. Please contact us with any A/B testing features that would add value to you and your company.

Last updated on