Automated Capacity Forecasting Using Predictive Analytics and Machine Learning

The Purpose of Capacity Forecasting

Every type of business process relying on information technology needs to be planned for. Whether the service is hosted in a public cloud or in a traditional datacenter, you will benefit from knowing how far your current resources will take you. Plus, capacity forecasting can better prepare you to take actions to sustain operation when necessary.

For most organizations, the primary driver for planning is to reduce the risk caused by poor performance or unavailability of the business services. By predicting such conditions, they can be proactively mitigated.   

Another reason for forecasting is to improve financial planning and efficiency. With forecasting, you can provide accurate and realistic performance data on which to base budgets and investment plans.

What is a Capacity Model?

A capacity model is used to predict future capacity needs and is derived from performance monitoring data of the components making up the service. In concrete terms, the model is an advanced mathematical formula that captures the characteristics of the service. It is used to predict how changes in activity or usage will impact the underlying resources. And if done right, it should be able to answer questions like:

• What happens if I increase the number of users of this service by 30%?
• Will there be any capacity issues in the coming six months that I need to address?
• What additional resources do I need to process 50% more transactions in the same timeframe?

To assist in this, it needs to both capture the key characteristics of the current service and allow you to manipulate those characteristics.

What are My Options for Capacity Forecasting and Planning?

A model can be constructed using several different methods, each with varying levels of detail and refinement. But in general terms, there are four type of models that are extra useful for capacity forecasting and planning. Below we explain the characteristics of each type and what a typical use case might look like.

Regression Models

This type of model uses regression analysis to make predictions about future behavior by making extrapolations from historical data. In its simplest form, it can do a linear extrapolation capturing an up/downward trend in a dataset. There are also more sophisticated variants that will account for seasonality, spikes and random variability in the data, making the forecast more refined and exact. Knowing when these peaks and troughs of usage may occur in advance allows you to scale up or scale down infrastructure, responding ahead of time to possible peaks in load, and saving money at expected quiet times.

There are a number of forecasting methods based on Machine Learning that captures the temporal dynamics of a time series, like Holt-Winters, ARIMA, Prophet, and polynomial regression. Another option is to disregard the time structure and base your forecasts on deep neural network architectures. But many prefer the former over neural networks since they can be computationally expensive and prone to overfitting. Time series forecasting uses the inherent order and structure of data to often provide quicker results.

Since this type of model operates purely based on historical time series datasets, it’s easy to automate the predictions. It is very useful for high level forecasts, where you need to understand the “big picture” rather than the details of each individual component.

Typical Use Case: A data center operations team needs to make a fair six-month prediction across their environment to identify capacity situations that will require extra attention or action in the coming period.

Correlation Models

A correlation model is driven by the relationship between one or more business activity metrics—often referred to as forecasting units—and a set of performance metrics. The correlation coefficient between the forecasting unit and a set of relevant performance metrics is established and can then be used to predict the maximum throughput of the service.

A correlation model starts with identifying the proper forecasting unit and making sure it’s captured together with your other performance monitoring data. Correlation analysis provides a “business-aware” model that can be automated to forecast service throughput.  

Typical Use Case: A service owner needs to determine if the existing resources assigned to the service can support the business demand changes forecasted for next year.

Queueing Models

This type of model is built on a combination of performance monitoring data and a detailed specification of the system being used. A queuing model offers the ability to evaluate a range of different what-if scenarios – including changes in transaction intensity, type of hosting platform or resource assignment. It also offers the ability to modify the model in order to handle a predicted issue.

A queueing model offers great granularity and flexibility, but the level of sophistication comes with requirements for detailed configuration and definition of a planning scenario which makes it less ideal for mass evaluation and automation.

Typical Use Case: A detailed plan for a specific business scenario that includes non-organic growth or multiple changes in parallel. It can also be used for closer examination of a prediction originating from one of the other modeling techniques.

Probability Models

This type of model makes forecasts by evaluating historical data to calculate confidence intervals to establish the most likely course of events. Probability models are useful for relatively short predictions, normally from a day up to a week. They are designed to provide a quick, visual cue of an expected seasonal pattern and deviations from that pattern rather than a detailed forecast of expected performance.

Typical Use Case: Produce a list of systems or services where the observed behavior is outside (exceeds or falls below) a certain percentage of the historical pattern and action might have to be taken. Provide quick at-a-glance charts to establish the magnitude.

How to Determine What Kind of Capacity Forecasting Your Business Needs

We concluded earlier that a Capacity Model needs to be built on and driven by performance monitoring data. Since most organizations already are collecting monitoring data, you could potentially craft your own models from scratch without a specialized tool. But before you do that, there are few things to consider:

• Is the data readily available? Data acquisition and preparation can mean heavy lifting and take up much of your time. Time that is better spent analyzing it.
• Is the data detailed enough? Regression Models based on time series forecasting requires all characteristics to be present in the data set—trends, seasonality shift, peaks etc.
• Can it be automated? Capacity forecasting is a recurring activity with plans that need to be updated on a regular basis. Without automation you may struggle to keep up and to have a reasonable total cost of ownership (TCO) for the solution.
• What is the Data Science experience?  To build accurate models, you need to have a good understanding of what type of analysis and methods that are relevant and applicable. The number of options can be overwhelming.

Vityl Capacity Management, from Fortra, streamlines data collection and integration with capabilities designed specifically for capacity modeling purposes. It offers a high level of automation combined with tried-and-true methods for predictive analytics. VCM simplifies and automates data access, analysis, and reporting, making it easy for your organization to eliminate wasted spend and prevent problems before they occur.