Tomorrow may not be like today

This statement has probably never been more true in the modern era. Successful decisions must be informed by data in real-time. Trends may start and end in weeks or even days. Demand shifts may occur in weeks or even days with “the new normal” driving radical changes in our expectations of the regular seasons in business.

‍

Cut through the decision fog and adapt faster by putting real-time data to work at scale.

‍

Opportunities may arise in unexpected areas. Excess supply for one market may be “just enough” for another. The retail customer may be more open to purchase different goods and services from brands she trusts or from new brands that have a compelling offer. Firms may consider new suppliers when availability or fulfillment problems arise. Competitors may not be paying attention to the changing environment, creating frustrated customers willing to try new things and to work with new suppliers.

I believe that now is the time to expand the use of real-time data in decision optimization. Some industries such as payment, hospitality and ticketing are literally built using real-time data. Other industries leverage such data for real-time inventory display, pricing updates or rate changes. In many cases, though, such data may have a narrow (if important) use in the customer-facing web properties. There is much more opportunity, however, to help us resolve potentially conflicting signals as we steer through the mist that volatility often creates.

Fortunately for us, key trends in real-time fraud and anomaly detection have driven enormous investment in the tools and technologies needed to make real-time data accessible and useful. A Global Market Insights report published in August 2019 sized the market at USD 20B in 2018 - growing to US USD 80B in 2025. The investments in this space are immense; entire solutions have arisen in the last few years around the problems of real-time data acquisition, filtering and analytics. Since most fraud detection systems operate live and “in the transaction,” the tools and technologies that support the industry have grown to handle data at a truly immense scale, with decisions leveraging the data made in seconds - or even milliseconds. Thanks to these developments in the information security and fraud detection spaces, we can deploy solutions based on proven technologies that - almost by definition - are operating successfully at scales far greater than we would typically need.

A highly transactional business by definition conducts business through hundreds of thousands or millions of transactions over a reporting period. Are analysts and leaders assessing these interactions on a daily (or hourly) basis to spot potential supply problems or changing trends? We are often conditioned to filter odd results and anomalies using the “dirty data” explanation. The inventory system shows stock or capacity available but no sales occur in a peak period. At best, we might investigate this as a “front-stocking” problem where the goods or services are not accessible to the customer.
At worst, we might ignore the unexpected signal as a blip on the premise that we “wait for a later review.” But what if the sales data are accurate? Is there unexplained loss or wastage? Or has demand suddenly shifted? What if customers simply didn’t show up?

Complex businesses are often in a bit of what one might call a “decision fog” - making decisions with incomplete or imperfect data. Adding a wide array of data sources that complement one another can help firms navigate through the mist. Consider some possible cues that could help in our lost sales example above:

• Site entry/exit data (physical or virtual, as appropriate)
• Foot traffic/impressions
• Stock movement
• Abandoned sales
• Mobile (person) location data

Consider what we might call situational awareness. In the face of conflicting indicators, what other data are available to tell us what is happening on the ground right now? In the "conflicting signals" example above, we probably won’t know the root cause of our anomaly. If we can use cues such as mobile location or site entry/exit data to confirm the presence or absence of customers, however, we can form a better hypothesis and take action more decisively. With a central “decision console” that brings these data elements together, leaders can make inferences and probabilistic judgments faster to help the firm keep moving in the right direction. Whatever the situation, periods of high volatility in supply and demand require us to take some definitive action to calibrate and adjust our decisions.

I like to think of real-time data like a wonderful street snack: hot, fresh, and available just when and where we need it. Building a comprehensive picture using observations from all the areas available can give us confidence to make better decisions quickly. While we can also use this data to develop insights through more sophisticated modeling (a topic for another day), our decision console would emphasize live data with minimal processing.

Let’s recap. We are in a decision fog exacerbated by high volatility in supply and demand. Our models have conflicting inputs and may generate recommendations that are very different compared to period norms. Even worse, poorly designed models may be unstable meaning the results vary widely and wildly as inputs change slightly.

Our best defense in the heat of uncertainty is to enrich our situational awareness. With a wide range of signals to help us choose the best alternative in the face of conflicting or inconsistent data, we can move beyond the inaction often fostered by the assessment of “dirty data” or “weird results”. To build our decision console, we first need to make an inventory of the data we have, as well as data we must acquire from new sources. Then we need to acquire and stream the data, using classification, clustering, filtering and other data science techniques as needed.

Here is an example of such an inventory as we might apply to an omni-channel commerce business in retail, food service or entertainment:

In such a decision console, there is no one data element that stands above all others. Instead, it is the complete picture on one screen that helps us to confirm or to refute a hypothesis quickly and to make a decision. When conditions are rapidly changing, we must reassess frequently. As I’ve detailed in another post, I believe it is critical to build a decision culture where we not only review our decisions as conditions change but also build a process to systematize such reviews.

Incorporating more real-time data in decision-making is not a substitute for building and using sophisticated data models - not by even a little. From my perspective, decision-making should be thought of in terms of navigating and steering. Our strategies set our course, and our data models help us navigate the best course towards that destination. Adding real-time data into a decision console can help us steer around bumps in the road ahead as we navigate through the fog and will help us see that proverbial “Road Closed” sign sooner, so we can adjust our course and get to our destination on time and ahead of the others who also want to serve our customers.

Winning the trend means spotting the trend early with eyes and ears attuned to the activity, environment and sentiment data around us. Especially in times of high volatility such as now, winning the trend means driving a disciplined, consistent process that encourages regular decision review and a deliberate and regular effort to gather data and insights that challenge original assumptions. Use real-time data to cut through the fog and make the most informed decisions possible when the path forward is not clear.

‍