Financial Services Giant Optimizes, Virtualizes, and Saves Big Money

Customer explains how TeamQuest Model helped IT increase savings and manage resources

A financial services giant had been going through a series of acquisitions as well as rapid growth. As a result, IT systems were in a chaotic state. The Fortune 500 company used Vityl Capacity Management to optimize its computing environment, particularly its newly virtualized AIX platform.

Rapid Growth

A series of acquisitions had taken place over a twenty-year period. As this coincided with rapid growth, there had been no time for consolidation or system integration. In addition, the company had significantly ramped up its web presence by bringing a host of services online, including a series of interfaces to facilitate heavy application usage by a large broker network.

Amidst all this confusion, AIX had become the the primary technology platform. The company had initiated an aggressive transformation program that drove AIX physical server growth by an average of 200% per year. In 2004, there were 48 CPUs running AIX. By 2006 — 344. In addition, the company transferred some mainframe applications to AIX.

Rapid-fire expansion over a period of many years led to increasing hardware costs, increasing logistical costs and delays (a lack of expansion floor space, electrical/UPS, A/C, etc.), and the high cost of keeping up with the demands of rapid evolution of the business by attempting to keep project lead times down. As a result, project teams struggled to keep pace with business demands, and multi-million dollar projects were being delayed due to a lack of necessary computing capacity.

The IT team looked for a new approach to reduce AIX Total Cost of Ownership (TCO) and overall lead time to service. The AIX Administration, Capacity Management and Architecture groups saw the new features in AIX 5.3 and the IBM p5 system as an opportunity. This added greater virtualization capabilities and made it possible to allocate CPU resources in increments of 0.1 CPU.

While the p5 systems with AIX5.3 looked promising, there were concerns about how to manage the systems while keeping an eye on the overall usage, and how to manage a virtual environment without adding risk to the business. Without virtualization, every logical partition (LPAR) has its own CPU. Once virtualization is introduced, all the CPUs would operate in a shared pool. This would mean that all applications would use the same CPU as managed by the Hypervisor. But if the pool filled up, the applications could crash.

Production systems were on two p595 frames with 64 CPUs. This included two shared CPU pools — one per frame. Therefore, if one pool went down, half of the production systems would crash. IT looked for a way to ensure resource availability without the possibility of a crash, while preventing the different business units from requesting more and more CPU resources. If IT could organize the environment more efficiently, it could provide high levels of performance without having to continually add more CPU resources.

Necessary Upgrades

In December 2005, the company upgraded from two p595s with 64 CPUs each to four p590+ systems with 16 CPUs each. That enabled a move from two shared CPU pools to four pools — one per p590 frame. By acquiring faster CPUs, it gained some time. But at the same time, the company needed a real vision on how to better utilize these CPUs to cope with long-term expansion.

By moving into a more virtualized environment, IT had to implement a process to ensure complete resource availability. That meant finding tools to provide accurate metrics on the shared CPU pool usage and finding a way to better communicate the results to the management team.

The IT staff consulted with Gartner to identify best practices, processes and tools. According to the IT team, Gartner identified TeamQuest (now a Fortra solution) as having the leading tools with the best cost/benefit ratio. That led to the establishment of an AIX/Capacity task force to gather data on CPU pool usage using Vityl Capacity Management. In addition, the company began gathering information on all of its servers in order to create the AIX Capacity Planning model.

Communication and Reporting Needs

To facilitate project communication and enhance communication to management, it was necessary to be able to automatically generate reports and keep close track of the shared CPU pools in order to always know the sum of all the shared CPU LPARs.

Vityl Capacity Management solved this issue with the creation of a custom table that gathered the required daily data for reports. This provided one table for all reports with only the essentials included such as p5 frame name, LPAR name and type (Dedicated vs. Shared), CPU Avg and Max (at 10 minute intervals). As a result, the new report takes less than one minute to generate and is fully automated instead of taking many days. Forty LPARs can be displayed on one graph with both the maximum 10 minute CPU usage for the shared CPU pool as well as the average.

Reporting within the enterprise has now been streamlined using Vityl Capacity Management. Despite having as many as 100 simultaneous IT projects in 2008, this financial services company was able to manage and report on them without undue difficulties.

Results

The financial services giant consolidated servers onto AIX 5.3. By conducting extensive modeling as part of the capacity plan, the company cut down on servers and increased the number of LPARs by 100 — despite a reduction in server administration personnel from 10 to 8.

Vityl Capacity Management has also enabled the allocation of CPU resources with confidence. Beforehand, the company utilized a contingency of 20 percent of total CPU usage to avoid resource contention. Modeling demonstrated that this was unnecessary. As a result, IT actually routinely allocates the number of virtual CPUs beyond the actual number of physical CPUs in the pool. Previously, if IT tried to figure all this manually in Excel, it would have taken a week and would have been inconclusive. Now, it is possible to model with certainty and it only takes minutes.

In terms of savings, probably the biggest area is CPU purchase avoidance. Rather than continually adding more CPUs, it is possible to avoid buying more processing power by managing and allocating resources more intelligently. Other savings have been realized in a reduction in power consumption, less need for cooling, operating within the constraint of existing floor space, and lowering maintenance by decommissioning older servers. Project delays due to lack of IT resources are also no longer an issue.

Modeling with Vityl Capacity Management has enabled the company to be more aggressive in capacity planning. IT staff realized they were paying too much attention to peaks on the graphs generated by off-hour backup processes. Now they focus more on usage peaks to ensure performance is always acceptable. At the same time, they can also analyze those off-hour peaks and spread the load across a wider time interval.