Fluctuations in demand, especially for perishable products, are often a major budget leak because it is difficult to predict demand accurately, timely, and regularly. Some retailers report up to 40% of stock unsold, resulting in significant barriers to forecasting accuracy errors.
The implementation of machine learning demand forecasting in retail can address this challenge, answering the recurring questions more swiftly:
- How often does inventory need to be replenished?
- What are the required stock levels for each SKU?
- How to calculate demand for different selling points?
- How to manage inventory when anomalies occur?
In this article, we’ll share our experience with developing machine learning analytics modules for demand and sales forecasting. Based on MobiDev’s 6+ years in artificial intelligence development and 15+ years in general software development, we elaborate on frequent concerns about what those systems are capable of, and what you’ll need to develop one for your specific case.
Sales Forecasting VS Demand Forecasting
Sales and demand forecasting are closely related terms but use different sets of data. While sales forecasting requires only historical data generated through actual transactions, demand forecasting may also include weather reports, customer surveys, web analytics, social media scraping, etc.
Businesses-wise, it can make sense to start gradually with the development of a sales forecasting system, if there is not enough qualified demand data. However, for both types of forecasting systems, it’s essential to understand that they are vulnerable to anomalies or unpredictable situations. It means that machine learning models should be upgraded according to current reality.
Preconditions for Using AI for Demand Forecasting
If you perform demand forecasting in some manual form with the help of spreadsheets, you might already have a set of data that you use for calculations. So at this point, it already makes sense to think about automating the whole process with artificial intelligence if you have a number of preconditions:
1. DEMAND DATA AVAILABILITY
Even though sales data is the most available piece of information for working with machine learning forecasting models, demand forecasting requires more than just historical sales. If you already set up some processes and infrastructure to collect data on your internal and external factors, it can be used for machine learning easier than collecting it from scratch.
Additionally, if you already have historical data for multiple years from the past, it will allow you to start developing an artificial intelligence solution and already bring value. Predicting inventory level implies we need your specific sales, which can’t be sourced elsewhere or bought on a dataset market.
2. PREDICTION FREQUENCY
Different business types will require different frequencies of forecasting. Conducting analysis on a monthly or even weekly basis can be a challenging task in terms of manual forecasting.
In this case, it’s a no-brainer that using machine learning frees up the resources, because it suggests an automated pipeline that gathers data and provides demand forecasts for given periods. One thing to keep in mind is that, the shorter the forecasting period, the less accurate it will be since it’s impossible to generate enough data to cover all the required time changes.
3. MULTIPLE VENUE DEMAND FORECASTING
For entrepreneurs who own multiple selling points that are set across different regions, forecasting demand will require a lot of human resources. Since seasonality, geography, and competition will all be different.
Setting up a system that sources data scattered across multiple databases, and presents analytics through a single dashboard, reduces the cost of demand forecasting in retail itself. In turn, a unified analytical solution presents a 360 view of your venues, inventory and sales activities allowing more flexibility in terms of planning and inventory management.
Based on our experience building demand forecasting modules for our clients, the use of machine learning for demand forecasting is a large competitive advantage. Since your business obtains better visibility and frees the human resources for development activities, rather than focusing on operational things.
From a technical standpoint, creating forecasts for multiple venues in most cases doesn’t require developing separate models for each specific case. The same goes for forecasting each separate product because we only need a suitable set of data that we can aggregate and properly feed to the model.
5 Key Factors Affecting Demand Forecasting Accuracy
Now, speaking of the actual demand forecasting models, we need to discuss what external and internal factors can impact its work. Since the ML model will derive its prediction from past events, its prediction accuracy rate is unlikely to be 100%. However, by understanding the following things we can mitigate those risks on the stage of building the actual model.
1. PRODUCT TYPES AND MODELING ERRORS
The product type is an important factor to consider for the demand forecasting model. For example, for a perishable item that has an actual demand of 100 cases, the prediction of selling 90 cases is preferred over the prediction of 110 cases. Missing the sales of 10 cases is a better result than wasting 10 cases, even though the actual error is the same percentage.
2. SEASONALITY
For each product, the seasonality cycle plays a crucial role in predicting demand. For instance, if you sell tents for hiking, its seasonal growth in demand appears to be during the summer period, with peaks in a certain month in your area.
If we take the data for only 5 months for training the ML model, the prediction of the machine learning model won’t give accurate results, since we need a year’s data as a bare minimum, to calculate the seasonality.
For products that don’t fluctuate in demand seasonally, say forks and spoons, a three-month data will be enough to start training the model and producing forecasts. But it always depends on the item itself and other external factors like competition, or amount of holidays for a given period.
3. REGIONAL IMPACTS ON MODEL PERFORMANCE
Demand prediction models are strongly influenced by regional factors that include customer behavior and cultural determinants:
- Marketing campaigns may be regionally specific and have a different impact that depends on where a customer is located.
- Holidays may vary between regions, which might be a consideration for adjusting the model.
- Legal issues/laws may limit the use of certain data in different regions.
4. NEW COMPETITORS ON THE MARKET
Demand forecasting is a dynamic concept. The more competitors and product alternatives are present in the market, the harder demand forecasting becomes. The competition level contains sub-factors, such as the number of alternative products and competitors.
So, it is a very good idea to add this information dynamically to your demand forecasting model.
5. ECONOMIC SITUATION
The state of the economy influences businesses and demand forecasting models. To put it more bluntly: periods of economic decline are likely to cause lower demand for expensive products, though sales of low-priced goods may go up. Therefore, an economic situation as well as trends aren’t external factors and should be considered when building artificial intelligence models.
How to Start Forecasting Demand With Machine Learning
Now, let’s discuss the practical steps to implementing machine learning for demand or sales forecasting. So, before embarking on demand forecasting model development, you should understand the workflow of ML modeling. What follows offers a data-driven roadmap of how to optimize cooperation with software developers.
STEP 1. BRIEF DATA REVIEW
The first step when initiating the demand forecasting project is to provide the client with meaningful insights. The process includes the following steps:
- Gather available data
- Briefly review the data structure, accuracy, and consistency
- Run a few data tests and pilots
- Look through a statistical summary
In our experience, a few days is enough to understand the current situation and outline possible solutions. This is a part of data science exploration that comes with the involvement of the client to gather the data and provide the most relevant corporate storage.
This stage can be a part of the AI consulting process aimed at finding the best possible solution that syncs your business goals, tech capabilities, and market needs.
STEP 2. SETTING BUSINESS GOALS AND SUCCESS METRICS
Before coming to the stage of developing a demand forecasting solution, a software development team needs to agree with the client/business owner on the success metrics for the model’s results evaluation. Success metrics offer a clear definition of what is “valuable” within demand forecasting. A typical message might state:
“I need a machine learning solution that predicts demand for […] products, for the next [week/month/a half-a-year/year], with […]% accuracy.”
This statement example will help you to identify what your success metrics will look like. You are expected to consider the following information:
- Product Types / Categories
What types of products/product categories will you forecast?
Different products/services should be considered and predicted independently for most cases. For example, the demand forecast for perishable products and subscription services coming at the same time each month will likely be different.
- Time Frame
What is the length of time for the demand forecast?
Short-term forecasts are commonly done for less than 12 months – 1 week/1 month/6 months.
These forecasts may have the following purposes:
- Uninterrupted supply of products/services
- Sales target setting and evaluating sales performance
- Optimization of prices according to market fluctuations and inflation
Long-term forecasts are completed for periods longer than a year. The main purposes of long-term forecasts may include the following:
- Long-term financial planning and funds acquisition
- Decision-making regarding the expansion of business
- Annual strategic planning
- Better accuracy
What is the minimum expected percentage of demand forecast accuracy for making informed decisions?
Implementing retail software development projects, we were able to reach an average accuracy level of 95.96% for positions with enough data. The minimum required forecast accuracy level is set depending on your business goals.
Examples of metrics to measure the forecast accuracy are MAPE (Mean Absolute Percentage Error), MAE (Mean Absolute Error), or custom metrics.
STEP 3. DATA UNDERSTANDING & PREPARATION
Regardless of what we’d like to predict, data quality is a critical component of an accurate demand forecast. The following data could be used for building forecasting models:
When building a forecasting model, the data is evaluated according to the following parameters:
- Consistency
- Accuracy
- Validity
- Relevance
- Accessibility
- Completeness
- Detalization
In reality, the data collected by companies often isn’t ideal. It usually needs to be cleaned, analyzed for gaps and anomalies, checked for relevance, and restored. That’s why data science consultants can be involved at this stage.
Data understanding is the next task once preparation and structuring are completed. It’s not modeling yet but an excellent way to understand data by visualization. Below you can see how we visualized the data understanding process:
This visualization demonstrates data decomposition, extracting trends, and seasonal or other factors from input data. It’s divided into several graphs:
- The 1st graph is an original timeline (time series visualization)
- The 2nd, 3rd, and 4th graphs separately represent seasonality, trends, and noise for further analysis and forecasting
STEP 4. MACHINE LEARNING FORECASTING MODEL DEVELOPMENT
There are no “one-size-fits-all” forecasting algorithms. The choice of machine learning models depends on several factors, such as business goal, data type, data amount and quality, forecasting period, etc.
However, there are multiple approaches to that, and they are often combined under the hood of a single forecasting system. The following approaches are used for demand forecasting and proved to be the most efficient:
- ARIMA/SARIMA
- Exponential Smoothing
- Regression models
- Gradient Boosting
- Long Short-Term Memory (LSTM)
- Ensemble Models
- Transformer-based Models
The topic of using ML models in analytical systems is much wider and covers not only retail cases. So if you want a much deeper insight into various models and approaches, read our dedicated article.
STEP 5. PRODUCTIONALIZATION
Once we have the data, we can start the training, validation, and improvement procedures. This process is done iteratively until the model achieves the maximum desired results.
A forecasting model can be used in its raw form, presenting its predictions in some sort of table, or sending mails with analytics for a certain period. However, more commonly we would also develop a front-end part of the application which is basically a dashboard that presents insights and visualizations through a single interface. It allows the user to query different reports, share them with the stakeholders, customize visualization types, etc. After testing and approval, the product is deployed into the production environment.
One thing to mention is that to keep the forecasting capabilities up to date, the model will require constant updates with new data. These can be your daily transactions or inventory turnover. For this purpose, we recommend setting up an automated pipeline that regularly aggregates recent data and updates the prediction model accordingly. It will help to keep the reliability of the developed demand forecasting model on a regular basis.
Addressing Anomalies and Unpredictable Situations in AI Demand Forecasting
As the demand forecasting model processes historical data, it can’t know if the demand has radically changed. For example, if last year, we had one demand indicator for a certain type of consumer product, it can change next year due to economic turbulence, or supplier countries being cut off from exports.
In that case, there might be several ways to get an accurate forecast. Here are the five most common ways:
- Collect data about new market behavior. Once the situation becomes more or less stable, develop an AI demand forecasting model from scratch.
- Apply a feature engineering approach. By processing external data, news, a current market state, price index, exchange rates, and other economic factors, machine learning models are capable of making more up-to-date forecasts.
- Upload the most recent data and provide it with the highest weights during model prediction. The period of a loadable dataset might vary from one to two months, depending on the products’ category. In this way, we can detect shifts in demand patterns and enhance forecast accuracy in a timely manner.
- Apply the transfer learning approach. If there is any gathered historical data, we can use it to predict demand in the context of the current crisis.
- Apply the information cascade modeling approach. We can forecast how people will make buying decisions according to the behavior patterns of most people.
During AI app development, artificial intelligence engineers analyze historical data for forecasting. This forecasting cannot predict the disruption caused by a global pandemic, a war, or a cataclysm, for example. Such an event requires the future recalibration of the machine learning models.
But keep in mind that after the demand situation normalizes after the pandemic/war/ etc – you need to adjust your model back, since in other cases – the model can remember the pandemic’s pattern and predict it for the next short-time period (e.g. next year).
Now let’s take a look at our client’s success stories to learn how demand forecasting projects look in practice.
Success Story #1: Implementing AI Demand Forecasting for a Retail POS System
About the client:
Comcash is a US-based ERP and POS system provider for the retail industry. Back in 2013, the company’s CEO Richard Stack engaged MobiDev to rebuild the old product. Since then, the MobiDev team has been working with Comcash on the constant technical development of the solution, delivering advanced features to help the client achieve new heights in the market.
Project goal:
After turning the system into a cloud-based solution, we suggested Comcash implement the demand and sales forecasting modules to extend the POS functionality and gain additional competitive advantage.
How we delivered:
We implemented an adaptive selective model for AI demand forecasting that predicts demand for the coming weeks based on user retail sales data. The main challenge was to ensure a high level of adaptability of the model to allow users to obtain a forecast for specific products at any given time.
To do this, we used a combination of DS/ML libraries and methodologies such as Pandas and ABC-XYZ analysis.
Moreover, a statistical report was generated to classify products based on popularity and profitability to help retailers identify the most profitable and least profitable products. This created the basis for advanced sales analytics and planning targeted discount strategies.
Outcomes and achievements:
The increasing popularity of Comcash led to its acquisition by POS Nation in October 2022. Today, Comcash boasts a presence in over 3,000 locations and seamlessly integrates with a wide range of custom hardware.
[Together with MobiDev], we’re able to work on a 24-hour development cycle, and we release software repeatedly faster than any of our competitors — and there is no overtime. We could never create what we have with MobiDev in my office in California. The tech market is just too competitive these days. If you are interested in developing a world-class product and working with a great group of friendly co-workers every day, I wholeheartedly recommend MobiDev.
Read the Full Success Story:
DEVELOPING A LEADING CLOUD-BASED ERP AND POINT-OF-SALE SYSTEM FOR RETAILSuccess Story #2: Building AI Demand Forecasting Module for Venue Chain Management Ecosystem
About the client:
Since 2014, MobiDev has been working with SmarTab, a US-based company providing POS systems for nightclubs and bars. SmarTab’s CEO contacted MobiDev to take over product development and improvement after his collaboration with another software vendor resulted in a pilot version of the app that fell short of expectations.
For over a decade of our joint work, the SmarTab system has expanded with new enhancements, including AI demand forecasting features.
Project goal:
After building the basic functionality of the POS system, the client was looking to enhance their product with an advanced dashboard allowing venue owners to analyze performance and make plans based on machine learning demand forecasting.
How we delivered:
To achieve the SmartTab project goals, we used a time series approach with a combination of Gradient Boosting and KNN models. The major components to analyze include trends, seasonality, irregularity, and cyclicity. With that, we can predict how much revenue will be earned within the next upcoming year with the daily granularity for each venue and total venue chain.
Users can view details on the product range of their venue in a few formats. This includes the pie chart, a 3×3 matrix, and trend-up/trend-down graphs to check the product’s popularity in a dynamic setting. This toolkit helps venue owners and managers make updates to the product range more effectively.
Outcomes and achievements:
Since the beginning of our cooperation in 2014, SmartTab has transformed from a bold startup to an industry leader that serves 700+ venues and chains. We continue working with the client on product enhancements, including improvements in demand forecasting models.
Read the Full Success Story:
DEVELOPING POS SOLUTION THAT REVOLUTIONIZES HOSPITALITY MANAGEMENTLeverage AI Demand Forecasting for Your Business With MobiDev
Once you decide to run a project on demand forecasting development, pay attention to the proficiency of a hired team. Demand forecasting projects fall into the category of machine learning and data science, which requires in-depth domain expertise for processing data and choosing the right approach to solve your specific problem.
MobiDev has been providing an all-around service for developing artificial intelligence products of different scales and complexity, customization options, and integrations since 2018. We have multiple projects in our portfolio and over 15 years of experience in software development.
Contact us below or book a call with a MobiDev expert to discuss your project needs.
