Crossing the great forecasting divide – make the transition with manual processes and AI/Machine Learning


It is true – forecasts from existing demand planning systems, built largely on historical data, will be significantly off, both during and after the pandemic. To quote Yogi Berra, “It’s tough to make predictions, especially about the future.” And right now, we are experiencing an event that hasn’t happened since 1918 providing very little historical experience to draw upon.

With so many dramatic changes in consumer behavior, retailers and consumer products companies who have been accustomed to a certain level of predictability, for their hero SKUs at least, are seeing even those categories swing dramatically. Consumers are buying more online; are bringing what used to be outside services into their homes with purchases like gym equipment and espresso machines; are applying a much more risk adverse mindset to their choice of items and sources; and are buying now with an overarching concern about the economy and their future employment.

How can retailers and consumer products companies triage during this volatile period? And what will be their new baseline on the other side? While some have suggested abandoning statistical models altogether and returning to judgement calls from human teams, the truth is somewhere in between. Three major steps can be taken that will marry the best of data and AI modeling with human oversight to help retailers and consumer products companies survive the changes and foresee the new normal as the pandemic resolves.

Pivot to a consumption sensing forecast model based on daily consumer purchase data. Daily consumer purchase data from different regions, like China and Italy, can provide early insight for other regions about lifecycle of demand before, during and after pandemic impact. Take automobiles, for example. In Wuhan, demand has recovered and surpassed pre-virus demand in under 5 weeks, fueled, perhaps, by a sense of concern about public transportation and virus exposure. This pivot requires digging into very different kinds of online consumer data from the increasingly powerful online marketplaces like Amazon. Health data about the timing of the infection curve can be combined with this purchase data to provide a clearer short-term view.

Leverage data and machine learning to do the heavy lifting. Daily consumer data inputs still need to be modified by differences in employment, CAGR, population density and other socioeconomic factors that will magnify or minimize the elasticities of demand from one locale to another. Machines are an enormous help in analyzing copious quantities of data, doing the heavy lifting to ensure that basic underlying factors are not overlooked. Follow the model of the Federal Reserve; instead of giving up on models to forecast the economy, they have augmented “slow data” with “fast data”, requiring their models to expand to incorporate, and learn from, even more near-term inputs.

Provide an interface that simplifies human oversight. Data, AI and machine learning, leveraged in this way, will help identify the needles in the haystack – the biggest anomalies in short-term consumer behavior that warrant human review to right the ship. Providing an interface to make the data easy to understand and easy to react to is more critical than ever.

In these changing times, “science-as-a-service” is the far better way to execute demand forecasting that is based on consumption sensing. Leverage data scientists to help capture the right signals and tune your models now so that a return to the more efficient state of automated demand planning will be faster and smoother on the other side. Near-term modeling will make it easier for planners to identify the most significant impacts requiring immediate adjustment, and the resultant machine learning will also benefit from tuning that separates near-term noise from longer-term, more permanent impacts.

Now is the time for a hybrid approach, blending data, machine learning and data science with human judgement to emerge victorious from this challenging pandemic.