Should you add AI testing to your tech stack?

Everyone’s talking about AI, but for marketers, that’s nothing new. Most enterprise marketers have been using AI to personalize for years. Businesses have made tremendous progress in consolidating their data in warehouses or CDPs to build holistic customer views which consolidate engagement, purchase, and marketing events. Centralized data, in turn, has proved fertile ground for machine learning (ML) models to personalize. As the AI marketplace matures, most “layers” in the martech stack have added ML capabilities. 

Marketers chasing the dream of 1:1 personalization are wondering if they should rely on the systems they have, or add another product to their stack. After all, most automation platforms – meaning the platform the marketer uses to orchestrate communication with their customers – now offer suites of AI tools. Is it worth adding a dedicated platform for AI personalization?

In this blog post we’ll answer three questions marketers should ask themselves as they consider adding to their stack:

1. What kind of personalization are you looking for? Traditional methods like “next best action,” which find “winners for segments,” can be done with predictive models (like churn propensity) or just with manual rules. Most modern automation platforms have these capabilities. On the other hand, marketers looking to make true 1:1 decisions – finding the best product, incentive, channel, message, creative, timing, and frequency for every customer – likely need a product which specializes in AI testing.

2. What kind of data will you use to personalize? Automation platforms typically have data for events like opens, clicks, and conversions. Marketers hoping to use all their first- and zero-party data to make optimal 1:1 decisions – including customer and industry-specific data – will need a more flexible platform which can personalize using all that data.

3. What are you trying to optimize? Automation platforms typically offer AI tools which aim to maximize metrics which the platform can measure directly – e.g. clicks, conversions. But conversions aren’t dollars. Marketers hoping to optimize financial metrics – like revenue, profit, CLV, APRU, or NPV – or to solve constrained optimization problems – like maximizing NPV less promotional spend – need an AI decisioning platform with the flexibility and robustness to optimize these metrics.

Let’s take a closer look at each of these questions.

What kind of personalization are you looking for? 

Let’s start with the first, and perhaps most obvious, question – what do we mean by personalization? Marketers and tech vendors alike use this word to mean nearly anything. To take an extreme example, addressing customers by their first name in an email could be called personalization – since after all, we are matching names to customers 1:1. In practice, marketers who say they are “personalizing” typically mean some combination of two things:

1. Predictive models – e.g. churn propensity, propensity to repurchase in a given category, customer lifetime value prediction. A churn model might give every customer a score on a scale from 0 to 100 based on how likely the model thinks the customer is to churn in, for example, the next 3 months. A category affinity model might predict how likely a customer’s next purchase is to be in a given category. 

2. Segments and rules. Taking certain actions for certain customers – that is, applying business rules to a given segment – is a form of personalization. These rules need not rely on AI at all. A marketer who designs a “welcome” journey for customers who have created an account is, implicitly, defining a segment and business rules – the segment is newly registered customers, and the rule is “send the customer this sequence of emails.” 

Traditional “next best action” (NBA) models combine these methods. The outputs of predictive models are used to define a collection of “if-then” business rules. For example, a rule could be: If a customer has a churn propensity greater than 75, then send them an offer for 20% off to renew their subscription. A next best action model consists of predictive models and business rules which are used to make recommendations.

These methods – manual rules, segments, predictive models, and regimes like “next best action” – offer limited personalization. 

1. Traditional NBA models typically make recommendations for segments, not people. They offer "personalization" which is not very personal.

2. The only way to know if NBA models are really making optimal decisions is through manual testing – usually A/B testing or multivariate testing by segment.

3. The “action” the model recommends is usually a product or offer. Traditional NBA models cannot simultaneously optimize all the decisions marketers must make with every customer touchpoint – e.g. message, creative, financial incentive, channel, time, day, frequency – much less handle the interdependencies between these decisions.

Marketers are beginning to realize that using predictive models and business rules to pick “winners for segment” isn’t getting closer to the dream of true 1:1 personalization. A newer technology, AI testing, based on a type of machine learning called reinforcement learning (RL), offers fundamental advantages over the older “next best action” approach.

1. AI testing models continuously learn and adapt, and automatically adjust to market changes.

2. AI testing leverages all first-party data about every customer characteristic to make 1:1 decisions about individuals, not segments.

3. AI testing empirically discovers the best decisions for each customer which maximize the marketer’s KPIs – no need for manual testing to see what’s working.

4. AI testing can optimize across multiple dimensions simultaneously.

Personalization can mean many things, but what marketers really want is to make the best 1:1 decision for every customer, and that means AI testing.

What kind of data will you use to personalize? 

Marketers who want the power of 1:1 personalization will naturally start with the tools they already have. Instead of adding a “layer” to their martech stack specifically for AI testing, why not simply use the capabilities of a marketing automation platform? Most modern automation platforms have extensive AI tools, including predictive models. 

Automation platforms typically do not have all of the data a marketer has about a customer. For example, an energy company likely wants to know how much electricity a customer is using; a credit card company may personalize based on spend in various categories. Marketers’ rich first-party data describes hundreds of customer characteristics, including data specific to their industry and their business. An automation platform is unlikely to have much of this data. A marketer might import the results of a few predictive models – like a churn score. But rules built on the results of a handful of predictive models reduces customers to only a few data points – the outputs of each model – letting the depth of first-party data go to waste.  As we’ve seen, predictive models can be used for traditional “next best action,” which picks winners for segments, but they can’t make 1:1 decisions for every customer.

If marketers want to personalize 1:1 using all their data, they will need a platform with the flexibility to use and decision on that data.

What are you trying to optimize?

Personalization, after all, is not an end in itself. Marketers hope to make 1:1 decisions for every customer because they want to influence customer actions. Many automation platforms have built-in NBA or “product recommendation” models, which predict what product a customer is likely to buy next. Suppose this model says we should send the customer an offer for jeans. What does that really mean? Here’s the most likely sequence of events:

1. The model includes category affinity models for various product categories. The customer’s highest affinity score is for the “jeans” category.

2. The model’s business rules say “recommend a product from the category for which the customer has the highest affinity score.”

In this case, the model is predicting that these jeans are the product the customer is most likely to buy next. But what the marketer wants to know is what action to take to maximize the metrics that matter to the marketer – in this case, perhaps CLV, repurchases, or revenue – and these actions likely are not the same. (For example, it’s possible that the customer would buy a different, higher margin product if the marketer offered it.)

Models native to an automation platform typically try to maximize metrics native to those platforms, like opens clicks, and conversions. But clicks aren’t dollars! There’s no reason why the action – or for that matter, the channel, message, sending time, or frequency –  that maximizes clicks would be the one that drives the most financial lift. The AI capabilities of automation platforms are typically not able to maximize financial metrics, must less the specific or custom metric most important to a particular marketer – such as maximizing profit while minimizing promotional spend.

Where does that leave the marketer?

Most enterprise marketers want to personalize by making the best 1:1 decision – product, incentive, channel, message, creative, timing, frequency – for each customer. They want to make these decisions using all their first-party data, and they want to make the decisions which maximize not clicks or opens, but their own financial metrics.

Making do with a platform whose primary capability is marketing automation is not likely to meet these goals. Because 1:1 decisioning is such a critical capability for enterprise marketers, we are seeing a new core platform emerge in the martech stack: the AI decisioning platform, which has the flexibility and robustness to give marketers the personalization they really need.

Nathaniel Rounds writes about AI and machine learning for nontechnical audiences. Before joining OfferFit, he spent 10 years designing and building SaaS products, with an emphasis on educational content and user research. He holds a PhD in mathematics from Stony Brook University.