Organizations seeking better business results from innovation often confront an apparent paradox: On the one hand, they can point to an impressive track record of innovations that contributed to their current market position. On the other hand, their people are unable to explain how, precisely, this innovation success was achieved. This incomplete picture of how innovation really happens makes it difficult to determine what’s working well, how to replicate it, or what could be improved.

The first step toward resolving this paradox requires recognition that, in most organizations, a great deal of innovation happens via informal mechanisms. Consider the case of an insurance company we worked with that had reached its top-tier market position, in part, by introducing a number of industry-shaping innovations over the years. One example they were particularly proud of was an early “mobile agent” app that enabled policy shoppers to interact via their phone with a sophisticated virtual agent powered by artificial intelligence.

This project started when a junior information technology (IT) developer, whose outside hobby was creating gaming apps, decided to spend time experimenting with a mobile agent app as a personal challenge. After developing a prototype, he shared the result with his manager, who was enthusiastic about the idea and encouraged him to continue. Shortly thereafter, the company chairperson visited their department, where he was shown a demo. The chairperson was so impressed, he pushed for it to be resourced with a project team, and from there, it followed the formal product development process. While the idea went through a series of well-defined stages all the way to launch, it was clear its genesis and early development — including early decisions to allocate time to it — were pure serendipity.