Five Things You Should Do Before Running Your First A/B Test
When we think about CRO, most people’s minds jump straight to A/B and multivariate testing. Sure, those are key methods used in the CRO process, but they only represent a fraction of what’s really involved in CRO and are more likely to take place after initial research and hypothesis building, than at the start.
Here’s five steps I recommend you take before committing budget and resources to carrying out A/B testing and CRO work proper.
1. Define your goals and any potential blockers
Here’s the thing, Conversion Rate Optimisation as a practice isn’t really well named. Why? Because it implies we should prioritise efficiency over being effective.
Yes, we want to make our website more efficient and drive higher revenues, but to do that requires us to be in alignment with the rest of the organisation: from marketing to product and development teams. Without a shared vision of what we want to achieve through optimisation and how it relates to the wider strategy, we risk running tests that have no real bearing on where the company or brand wants to be.
Take the time early on to get stakeholders around the table and get explicit about what you want to achieve through your CRO efforts. At the same time, get real about potential limitations such as resources. If you’re struggling with development time, for instance, this could be a blocker to making progress and so impacts how realistic your goals are.
Genuine Conversion Optimisation has to align with real business value, in other words, drive more revenue in a way that's more profitable. In most cases, this is hard to do in isolation from the wider business.
2. Get clear (and honest) about your assumptions
If you’re thinking about getting started with CRO for your website or app, chances are you’ve hit a problem of some kind and that’s what’s led you to this point. The problem might look something like this (these are real problems I’ve helped clients with):
Lead Generation:
“I run an accounting firm. We spend a lot of money on PPC to drive leads, but we just aren’t converting as many this year. We think our landing pages are the issue. We need help redesigning our landing pages so we can increase Conversion Rate”
E-commerce:
“We’re driving a tonne of SEO traffic to our product pages, but users just aren’t adding to basket as much as they used to. I think the layout of our product pages is too cluttered and it’s putting users off. I want someone to come up with a product page redesign so we can A/B test which one works better.”
SaaS:
“We’ve noticed a big dip in users converting from our freemium model to a paid subscription. Our Product Manager thinks it’s because we’re not showing users enough value quickly enough in the demo. Can we run an A/B test where we move some of the feature gates?”
In each of these cases, a problem has been identified - Conversion Rate has dipped in one way or another. However, along with that, two pretty big assumptions are being made - what’s causing the dip in Conversion Rate, and what action needs to be taken to rectify it. But assumptions like this are risky.
Reduce risk by working through your assumptions
Though useful in early discussions around optimisation, we can’t rely on them as a source of reliable insight. They’re too bound up in various kinds of internal bias and usually aren’t the result of rigorous research. But it’s still important we get these hunches out into the open and recorded in some way so that we know what we’re dealing with in terms of what we know, and crucially, what we don’t know.
Before investing time and money into running costly tests, we want to get real about what we know and don’t know. A great way to do this and galvanise your team around a shared understanding is to run through the following matrix and get real about where the gaps in knowledge are.
Credit: Stephen Puiszis
Once we work through each of the assumptions above, they might actually lead us to some useful jumping-off points to frame our research questions and get more clear about what we need to know:
Has the composition of landing page traffic changed over time?
Do users scroll far enough to see critical content, such as our free shipping information or product reviews?
Which features in our demo are most correlated with users taking out a subscription?
3. Learn the basics of UX research methods and when to use them
Before even getting close to running a test, it pays to develop a basic understanding of user research and the full range of methods and methodologies available to us.
Attitudinal vs Behavioural Research
Learning the differences between quantitative and qualitative research methods, and when to use each, is really where you should start. This matrix provides a really useful attitudinal vs behavioural way of looking at research methods to help you land on a method that makes sense.
There are plenty of fantastic resources you can turn to if you’d like to learn more about the basics of UX research which are all available for free.
Here are some helpful resources to get you started:
4. Actually talk to (and observe) your users
Related to my third point, I can’t stress enough the importance of actually talking to your users. Even if you’re pretty inexperienced in user research, having a discussion with a new customer about what they’re trying to achieve, what their pain points are and why they chose your service is significantly better than doing no qualitative research at all.
When it comes to CRO, we’re working with websites and apps that are already in existence and being used by customers and potential customers. That means we can go a step further than just talking to our users, and take the time to observe customers.
Why is this so crucial? Because this kind of observational and behavioural research tends to give us really solid insight into why users struggle to complete tasks, such as completing a checkout process. Without this qualitative ‘why’ insight, we’ll always be in the dark about why issues are occurring.
5. Get close to your analytics data
With an abundance of free and low-cost digital analytics tools on the market, there’s really no excuse for brands not to be familiar with how users actually use their products.
While there’s a range of tools om the market, all share similar characteristics in that they all collect data on who users are, what they are doing on our websites, and where they have come from. If we combine these pieces of information together, we can start to piece together a theory as to why something is happening.
It’s this why that’s so powerful. It can help us identify weaknesses in our website, whether that’s landing pages that don’t lead to high-quality conversions or leaky purchase flows. Qualitative data can’t really give us this kind of insight, but digital analytics tools like Google Analytics most certainly can, and often for free.
Moving forwards with CRO and testing
Once you’ve completed the above steps, you’ll have gathered critical information around your business, limitations, customer needs and existing problems. These are all key pieces of information that are going to help you generate hypotheses and test scenarios that make sense for your business and are actually do able.
If you want to talk to an experienced CRO consultant about improving your Conversion Rate, get in touch for a free discovery call.
