Skip to main content

A / B Testing

To improve user experience, Atomic enables A/B testing of cards, and then the ability to analyze test results quickly and easily.

How you achieve this will depend on your preference and resources available to you. There are four suggested patterns:

Events filter​

Creating an events filter is the recommended pattern, as it makes the analytics easy to compare between card versions.

It requires your engineers to build a filter for your events before sending them to Atomic. The filter would randomly choose a card to trigger, and send the corresponding event to Atomic.

Set up an events filter​

  1. Ask your engineers to build a filter to randomly segment your trigger events, before they are sent to the Atomic API. This is ideal when you are handling A/B logic upstream from Atomic in a marketing automation type system, or wish to control the A/B behavior.

  2. In the Workbench, author a card template (A). See: Author a card template, and take note of the API trigger URL for this card (on the Triggers tab).

  3. Duplicate the card template to author the B version:

    • from the Cards list screen, in the overflow menu for the Card you just created, choose 'Duplicate card', then locate and open the new card by switching to the Drafts tab, or
    • from the Cards list screen, in the overflow menu for the Card you just created, choose 'Duplicate card', then:
    • author a new card by going to ‘Cards’ and then ‘New card’
    • paste the ‘Raw card data’ into the new card.
    • don’t forget to publish the card. See more on publishing in managing cards in Atomic

  4. Edit the B version to match your test requirements. This could be any compilation of layout, media, action buttons, or copy etc.

  5. Take note of the API trigger URL for this second card (on the Triggers tab).

  6. Publish the cards, and then have your systems send the branched versions of the event to a different trigger: for the A or B versions.

Segments​

Using Atomic's customer segments features you can define cohorts of customers using Segment filters, then use segment-entered triggers to deliver different card variations to customers when they enter or exit the segment.

Segments are created using filters that look at customer profile values, recent analytics events and tags. Tags are an especially easy way to identify groups of users, because they can be added when manually importing customers via a CSV file or API. If you are adding/updating user profiles by syncing records from an external system, you may also be able to use custom fields from your imported data to build segments.

How to use segments in A-B testing​

  1. In the Workbench, define a segment in the Customers area of the Atomic workbench by adding relevant filters, then saving the filtered view as a segment. Repeat this for each test group you will target.

  2. Author a card template to send to each segment in your test. See: Author a card template.

  3. For each card template you create, add a segment-entered trigger from the Triggers tab within the card editor and choose one of your segments.

  4. Publish the cards. During the publish process, you may be prompted to decide whether any customers who currently match your Segment's filters should be sent a card. In most cases you will want to do this. After the card templates are published, any customers who enter the relevant Segment will be sent the card.

One-off split testing

If you want to test different cards as part of a one-off card send, you can adapt the steps above. Instead of adding segment triggers to your card templates, you can simply publish the card templates. Then, from the Customers area of the Workbench, browse to each Segment, choose 'select all', then follow the steps to send a card manually. This will send the chosen card to the selected users from the segment. Repeat these steps for each variation you want to test.

Manual switch​

Rather than running A/B tests concurrently, you can manually switch the card at a specific time, and then segment the analytics data using time filters.

The length of time to run each segment will be determined by your sample size and level of confidence desired in the results.

Set up a manual switch​

  1. Pick a time period to run the test that minimizes the effect of other variables. E.g. control as best you can for day of the week, time of day, usage of your app, and any other significant events for your customers and company.

  2. In the Workbench, author a card template (A). See: Author a card template and a corresponding card trigger.

  3. Prior to the agreed switch over time, begin making changes to the card for the B version. Note: changes to the card will not be sent to customers until the card is re-published. See more on publishing in managing cards in Atomic.

  4. At the agreed switch over time, re-publish the card with the changes desired for the B version.

  5. Once the experiment is complete, segment the analytics data based on the switch over time/date.

Variable switch​

Card copy

This method is only suitable if you’re A/B testing based on differing card copy. For example, what is said in the headline, body, or action button. You can not use this method if you wish to test different versions of media, or the number of action buttons used on a card.

Atomic provides a mechanism to add metadata to events, as the detail of any variable in a card is included in analytics event payloads.

For example, if you add a variable called copyVariant to a card, then in downstream analysis, the version of the card would be identified and segmented by the value of the copyVariant variable, which will be included in all the related analytics event payloads.

Set up the variable switch​

  1. Author a card template. For this method we use the same card template and trigger for both A and B versions. See: Author a card template and card triggers.

  2. Add a variable, for any copy you wish to test in your A/B test. E.g. the copy on a button, or within the headline. See: Add a variable

  3. When your engineers send events to Atomic, the payload contains a different version of the variable.

  4. Segment your analytics based on the variable used (returned in the analytics payload).

Analysing A/B Testing results​

For the events filter and manual switch methods, the - Insights area provides the quickest and easiest way to analyze your data. Here you can easily segment the data based on date, stream, or platform.

Rich analytics data to plug into your existing analytics platforms is also easily available through either batch download, or API. See more about getting analytics data from Atomic at: Analytics from Atomic.

First class feature

We’re currently thinking about ways to support A/B testing as a first class feature in Atomic. Please contact us with any A/B testing features that would add value to you and your company.

Last updated on