How to run A/B and multivariate tests on accounts, households, or other user groups
Most testing platforms only work at the “user” level.” This playbook will show you how to run A/B or multivariate tests on any object or user group, such as at the account or household level.
Made by: Hightouch
/
6 minutes
Most experimentation tools assign each individual user to experiment variants. Unfortunately, many businesses operate on other entities or user groups. Let’s say you’re on the product team at a B2B SaaS company. You want to test an alternative onboarding experience for new trial signups–ensuring your users get to that ‘AHA’ moment and convert to paying customers. Unfortunately, since your usual experimentation tools create split audiences at the user level, different users from the same account get disjointed onboarding experiences. Your customers are confused and your test results are invalid: you can’t tell what’s working to convert trial accounts to paid!
You need the ability to create a split experiment at the account level, ensuring that all users from an account get the same experiment variant. Similarly, a utility provider might need to split experiments at the household level. Unfortunately, A/B and multivariate testing based on anything other than a user can be difficult or impossible with many experimentation tools.
In this playbook, we’ll demonstrate how to use Hightouch Traits to set up experiments that group users however you need to, so that members from the same household, account, or other user group get the same test variant. We’ll create a deterministic random variable trait, and assign all members of the same grouping (such as account) to the same trait. This allows us to:
Build audiences segmented by the assigned variant for the account
Send the account’s assigned variant as a user property directly to downstream tools
Write the account’s assigned variant back to the data warehouse for data science or other reporting
Bonus: use the trait directly within Hightouch’s Campaign Intelligence for closed-loop reporting
You have connected Hightouch to your data warehouse
You have Customer Studio in your Hightouch account, which has the Trait feature discussed in this playbook.
You’ve connected Hightouch destinations for your treatment audiences. In this guide, we’ll use Iterable for email campaigns and Appcues for onboarding flows.
To build audiences and run split tests in Hightouch, you must ensure Hightouch can access your users in your data warehouse!
Hightouch’s schema builder makes it easy to set this up. You’ll create a “parent model” called users. If you have a “users” table in your data warehouse, use the “table selector” to use the entire table. If you don’t have just one table with the information you need, you can use SQL to build a “users” table within the Hightouch UI on top of your warehouse data.
You should also make sure this “users” parent model has a column for whatever grouping you wish to use in your experiments. In this example, we’ll use an “account_id " to group all users in the same account.
In this step, we’ll create a new trait for every user in our parent model. By following these steps, you’ll segment all your users into however many test variants you want and ensure that all users with the same account_id experience the same variant. Furthermore, this method will ensure that in future net-new experiments, account variants are re-randomized, so that the control accounts in experiment #1 are not always the control accounts in experiment #n.
In Hightouch’s left navigation, select Traits, nested underneath Customer Studio.
Click “Create” to make a new trait.
Select your “users” parent model, and select SQL Formula as the calculation method.
Customize the SQL statement below for your use case.
Replace ‘Exp1’ with a hard-coded string or experiment number that you’ll use to identify this particular experiment. You’ll change this in future net-new experiments to re-randomize your variant assignments.
Replace ‘3’ in the MOD function with the number of variants you need
After pasting the code in the Hightouch UI, use the SQL editor's autocompletion feature to replace the “AccountID” string with your reference field
Select Property Type = Number
Save your trait. Now, every row in your parent model has a numerical trait assigned to them that you can use to group them into experiment audiences; in our example, all users who belong to the same account are in the same experiment variant.
MOD( --the modulo function assigns the variantABS( --ensures the hash is non-negative
HASH(
CONCAT('Exp1', {{"AccountID"}}) --Exp1 is a hardcoded prefix to ensure the account does not get the same assignment for multiple experiments. Replace the "AccountID" string with your Account_id variable.
)
),
3-- the number of variants (including a control)
)
Like most experimentation tools, Hightouch’s native splits feature works best on individual users. To leverage our new custom trait and group experiments by account, we will build a separate audience for your control and each experiment variation.
In the navigation, select Audiences.
Start building your audience from your “Users” parent model.
Select a filter using your trait property. You can type to search for it or click on “Custom Traits” to find it more easily.
In this trait property filter, enter the value of your control variant (e.g., 0)
Preview your audience to confirm your user count.
Save your Control Group. Depending on your experiment setup, you might use this control group to exclude from all test audiences and/or report on it as a baseline.
You’ll follow the same process as Step 3 for each of your treatment audiences. The only difference is that you’ll enter the value of each experiment variant (e.g., 1, 2, or 3) for each treatment audience. If you want to segment by any other additional filters, such as geography, you can also add those now.
Now that you have audiences for each experiment, it’s time to take action! Hightouch can send these audiences to hundreds of different tools. In this example, we’ll use a prebuilt sync template to send each audience to Iterable with just one click so we can trigger different emails for each audience.
If your organization relies on user groupings, you need to ensure that members of the same group receive the same experiment variation. Like so much within Hightouch, this is easy to do because the platform is flexible: it adapts to your data, not the other way around.
If you have questions, feel free to schedule a time with Hightouch. A member of our solutions team will be happy to help you set up your own A/B or multivariate experiment.
