Keeping traditional design and development practices unchanged may not be the best path to success today. Decision makers today need more data - from many more sources - and they often need it in real-time. In the "Crawl, Walk, Run" methodology that most of us follow for enterprise software implementations, we tend to start with a representative sample of data along with a concept of how to generate the insights and support decisions through process flows. Stakeholders sign off on the prototype and then the design / implementation team takes the solution to scale.

‍

Unfortunately, this last scaling step is often problematic. Misunderstandings about the nature of data at scale, system performance issues, batch windows, infrastructure capacity issues and other factors often do not appear until the very late stage of a project cycle. Fixing these problems may involve eliminating data, simplifying models or cutting away some use cases just to “make it work.”

While reasonable in the heat of the project, these schedule compromises may reduce the value of the solution greatly as deadlines, budget and pressures to deliver begin to weigh on the teams. A hollow victory often emerges in these cases when the new solution is functional but doesn’t provide much of the needed benefits. Users lose enthusiasm or harbor much higher skepticism on the insights after seeing issues when they appear late in the project. While the teams counter these issues by planning enhancements in later versions, the business overall may not recognize (much) value until much later after one or more of these (originally unplanned) updates.

In the worst cases, the organization may lose sight of the original goals as business conditions change. With demand and supply volatility likely to be at historical peaks for some time, the business conditions may change so significantly before a new system can be built and released that a "re-think" is required. Consider this little axiom I have found to be true (it is a tribute to Yogi Berra for US baseball fans):

The longer it takes, the longer it takes

‍

Delays tend to compound on themselves. Key premises or attributes of our prospective solution may only be valid for a limited time. The root cause of a particular delay may indicate there are other effects that are not yet known. At a certain point, we have to re-think parts of the solution. "Re-thinks" almost always means more delay.

Rinse and repeat.

We think innovation needs an overhaul.
So what to do?

One key defense against this problem in decision support and optimization problems is to focus on scale first.

Success can come faster by "front-loading" the data cleansing and scaling problems. Rather than finding out a few weeks before a go-live that our storage infrastructure is undersized, or that or our databases can't ingest data fast enough, instead organize projects to prove the data is usable, accessible and clean - at scale - before we get into decision automation and workflow changes with users.

Run to Scale

If we make a small adjustment to our trusty "Crawl, Walk, Run" methodology, we avoid the problems of trust in the outcome by building trust in the data and in our ability to deliver consistently first. Such an adjustment doesn't have to mean postponing benefits or taking longer - far from it. We can employ real-time data streaming and semi-structured data handling techniques to deliver insights quickly. As the rest of the project moves ahead, increasingly valuable insights emerge as the system comes together. When users have experience with the real data early, we focus our trust and consensus-building on the system, the decisions and the outcomes.

Without trust in the data, users will not embrace a new system. If users lose trust late in the cycle, the effort and time to regain that trust compound to create (longer) delays in the program. Why take unnecessary chances? Adding resources, focus, and checkpoints early for data quality and scale will make the project more likely to deliver on the planned benefits when needed. In the most complex cases, you may want to consider bringing in a specialized data integration team to coordinate efforts with vendor and client.

Consider in your next project how can "run to scale" and help users benefit from new data sources even ahead of the rest of the system, while de-risking your schedule and building trust in the outcomes to follow. Explore how you might deliver reporting or intermediate insights "for real" to drive closure sooner on the quality and scale validation. Look at integration opportunities with other systems that are low-risk and valuable to stakeholders - areas where your newly packaged data or partial implementation can help solve related problems, again while building stakeholder trust. You may want to consider sharing data quality monitoring with stakeholders to learn about other "hidden problems" that might exist elsewhere in the landscape, avoiding future problems. Focusing on data scale and quality early will not only ensure you deliver value earlier, but also that you reach the final integration phases sooner as users have more time to build confidence in the supporting data.

At 3 Leaps, we help highly transactional businesses to automate, scale and speed decision-making. Follow us on Twitter or LinkedIn, or check out the rest of our blog posts.