I was motivated by a post on Reforge called Good Experiment, Bad Experiment in relation to growth, which was inspired by a classic post from Ben Horowitz on Good Product Manager / Bad Product Manager.
As a company, we’ve been running product experiments for a long time, and more recently, we’ve started doing more “digital growth experiments” with our clients. Digital experiments are still growth experiments but focused on larger organizations who struggle with moving from idea to product or service quickly.
Here is what I’ve learned about running digital experiments with our clients.
Good digital experiments are formally communicated as tests. Traditional organizations announce their strategic direction and new product offering without any success data to back it up. It’s basically: “We think this is the right way to go, and you should believe us.” And in larger organizations, if there are failures, the word travels fast, and it usually means someone’s getting fired or the failure triggers a change in leadership. Experiments are a set of tests and should be communicated as such. A failure isn’t just a failure. A failure should demonstrate lessons learned and should feed the next experiments. Bad experiments are treated as if a career depends on their results, which brings me to my next point.
Good digital experiments are run by dedicated innovation groups. If the company is large enough, a separate innovation group (with its own budget) should be running these experiments. This sets the tone within the company that the innovation group will definitely have failures and lessons learned. A failure doesn’t mean a company isn’t innovating; it means it’s trying out different avenues of innovation. Bad digital experiments aren’t structured and don’t have a dedicated innovation group with dedicated capital or resources. Bad digital experiments struggle with internal communication on why experiments are run, with many groups refusing to put their names on the line for innovation’s sake.
Good digital experiments can be run by anyone in the company, even if they aren’t part of the innovation group. The reason Flamin’ Hot Cheetos exists is because a janitor at Frito-Lay pitched it to the senior executive team. This was all because of an “innovation was everyone’s job” mentality set by the CEO. Good pilots allow everyone to pitch an idea and get feedback on it, as well as give the time and resources to test out the experiment. Bad digital experiment culture is created when only people within the innovation group are allowed to execute experiments. It’s certainly not the way to a truly innovative experimentation culture at all.
Good digital experiments fail quicker and without fanfare. Experiments that quickly prove or disprove a hypothesis, or within the defined time period, are the best type of digital experiments. Bad experiments are drawn out activities that everyone in the organization is counting on to save the company. The longer the experiment runs, without any movement (good or bad), the more likely a bad connotation will be associated with the experiment as a failure.
Good digital experiments are gated by success metrics. I’ve discussed this in a previous article about how we run experiments with our clients, and our experiments are almost always “gated.” This means that for experiments to become full-fledged core service offerings or products, they must first meet the metrics set in the first few tiers. Once they pass those gated success metrics, they can become a full offering with full resources and capital to run them. If they don’t pass the first tier of success metrics, they don’t move on and are documented for lessons learned. Bad experiments involve setting the expectation that a first-tier test should demand full resources for becoming a product or service that the company is offering.
Good digital experiments aren't always technology-focused. Sure, testing out Voice as a channel is a good experiment. But not every test should be a disruptive technology. It could be testing out a new packaging concept that improves the customer experience or a new customer onboarding process. Bad digital experiments assume that cutting edge emerging technology like Artificial Intelligence or Voice are the only things that can be tested.
Good digital experiment programs turn a percentage of their experiments into full-fledged service offerings or valuable lessons learned. After a certain period of time, there is an expectation that an experiment breaks its way through the gated process and becomes a real product or service offering. It either becomes a real offering or the lessons learned lead to a better-refined product. Bad digital experiments have no documented lessons learned or provide no new service offerings as a result.
Running growth digital experiments in large organizations is a cultural change. It’s not a “move fast and break things” mentality that Facebook was famous for. Experiments within large organizations are best run within a structured framework that is communicated to the organization. Instead of betting the farm on digital offerings, it’s best to take multiple bets, and the bets that win are communicated to the organization where they can be expanded upon.
Let the failures fail without fanfare, document your lessons, and move on to the next experiment.