Stages of the optimization journey
One of the most asked questions we get is “how do we mature our testing program?” There are a lot of variations of this with questions related to if a specific tool is necessary at the current stage, or what the team structure should look like. So I thought it would be useful to share a simple heuristic which helps you understand the stages (as I see them) that teams have evolved through on this journey.
I have worked in digital agencies focused entirely on experimentation for the last five years, and spoke with experimentation teams at dozens and dozens of organizations.
The thing I wanted to share up-front- this is a never ending journey and even the most sophisticated digital native FAANG companies have areas of their business which could use better developed experimentation programs. So don’t feel bad about where you are.
In fact, the less mature your experimentation program, the better the win rates of your experiments and the cheaper and easier it is to progress the programs. So jumping ahead to more sophisticated stages often is more expensive and gets less results than fully taking advantage of the opportunities where you are.
Let’s dive in.
Signs this is you: You don’t have any test tools.
Don’t feel bad. You have lots of company. This also means you have many improvements you can make when you start experimenting.
Ad Hoc testing
Signs this is you: Most of the experiments your team have run are to switch banners, or show a new experience to 100% of traffic.
Sometimes the data and the results of an experiment are ignored and a change is rolled out anyway because it is “strategic”.
This is an exciting stage. Often you see an awesome financial results from an experiment, which gets the organization motivated and ready for the next stage.
We have some smart strategic people working on experimentation in a centralized team
Signs this is you: Everyone knows who to go to talk to the experimentation team, and they are respected.
At this stage, processes are getting more developed. More strategic issues are the subject of experiments. A strong competent cross functional experimentation team of strategists, analysts, UX and research people are all bought into experimentation as a way to learn more about the customer.
The opportunity at this stage is much of the rest of the company is operating in the old way. Creating a business case for a project. Developing it. Launching it. Then squinting at the data until it looks like maybe it was successful. Repeat.
I feel free to cast some stones here, since I lived the first 10 years of my work life like this
Experimentation is just the way the company validates ideas, separating good ideas from the rest. Rather than a pocket of experimentation happening, it happens all over. Merchandising testing assortments. Customer Service testing canned emails, Marketing testing campaigns.
There are a lot of cultural factors that need to happen to get here. Like taking steps to reduce the Power distance, encouraging people to ask for forgiveness instead of permission, expecting a bias to action, and accepting failure as long as people are feedback hungry.
If the average person got a flat tire on the highway, it might take them 15 mins to change their tire. For a NASCAR pit crew, it might take under 15 seconds. How can you make your team a NASCAR pit crew?
As experimentation becomes more widespread in the company, the next stage is to dramatically reduce the time and effort it takes to run experiments. As many steps as possible are done beforehand, templatized, or automatically triggered. High quality experiments with good data can be created in minutes.
This requires an investment in release systems, data pipelines, and your tools- but reducing the number of steps and total effort is the only way to run 10,000 or 50,000 experiments a year.
Where do you think your company is? Let me know