TekSouth Fights US Air Force Datacenter Sprawl with Fusion ioMemory Solutions

The Challenge

TekSouth is a privately owned systems integration and professional services company headquartered in Birmingham, Alabama that services both commercial and government sectors. The US Air Force’s (USAF) Commanders’ Resource Integration System (CRIS) is its Authoritative Data Source (ADS) for financial management of unclassified appropriated historical data, and uses TekSouth’s end-to-end decision support architecture. CRIS integrates data from geographically dispersed financial managers and gives them the tools to manage day-to-day operations, while providing senior leaders at all levels a near real-time snapshot of how operational entities are performing.

Mike Rhodes, TekSouth’s VP of Operations, described the challenge CRIS faced: “We’ve been the technical prime for the USAF Financial since 1998. Every five years we do a technical refresh of hardware to improve performance. In the process of the latest rotation we wanted to reduce power, cooling, and overall footprint cost.”

The challenges CRIS faced were daunting:

1. Deliver near real-time reporting for more than 15,000 users running up to 1.2 million queries per month.
2. Perform Extract, Transform, and Load (ETL) jobs to prepare data for end users without affecting end-user application performance.
3. Address increasing pressure for datacenter space and energy conservation.
4. Meet the USAF’s failover requirements to be operational to users and managers in every time zone around the world.

The Solution

After investigating several technologies, TekSouth determined that a system using Fusion ioMemory was the best solution.

Scalable Performance for Ad-Hoc Reporting

The USAF financial warehouse posed significant challenges to scalable performance:

• Size. The warehouse contains over twenty-two terabytes of data and continues to grow, receiving 250 data feeds daily from 19 systems worldwide.
• Security overhead. Strict access controls protecting sensitive financial information added to the typical workload of standard data warehouses.
• Ad-hoc usage patterns. While most data warehouses primarily use canned queries and stored procedures, the vast majority of queries on CRIS are ad-hoc, yet require near real-time responses.

Mike states “CRIS supports multiple-terabyte table processing that reaches up to billions of rows deep. Typical user patterns are nearly 98% ad-hoc, which means we can’t optimize for canned reports.”

He then explains how moving the warehouse data onto Fusion ioMemory solutions addresses the problem, “We’ve designed the software CRIS uses to deliver the performance and high uptime the DoD needs. But we were limited by the capabilities of traditional hardware. With disk I/O being the biggest bottleneck, Teksouth’s architecture is built to scale out and isolate I/O intensive processes from one another. Fusion ioMemory eliminates the I/O contention that is the biggest constraint.”

The results were better than he hoped. “We tested the system against real-world historical query workloads rather than synthetic benchmarks, which allowed us to see how the ioDrive cards would perform in actual deployment. A single server with ioDrive cards doubled the workload capability of a 3-server, 21-disk array-based system. At 15 times the standard production workload, we were still operating within the USAF performance requirements,” Mike said.

Mike notes what this performance means to the USAF: “The new system supports three times the number of concurrent users and can run three times the number of queries in the same time. 82% of ad-hoc queries returned in under 10 seconds, while the average response time for all queries, in total, was less than 23 seconds.”

Escalating ETL and Availability

In addition to responding to end-users’ ad-hoc queries in near real-time, CRIS runs ETL jobs to prepare data for user consumption. These jobs run as often as hourly, and cannot slow system responsiveness, which would interfere with end users’ productivity.

A disk-based system would require massive overprovisioning to avoid resource contention and system slowness. The Fusion ioMemory system’s low-latency transactional performance enabled a single server to handle both active queries and ETL loads. “We run ETL jobs on a separate server, but this server is a failover for the data warehouse. The USAF requires the warehouse support double its standard query workload, which means that the failover system needs to support both the ETL workload and double the standard query workload.” Mike states “The ioDrive system didn’t begin to see any degradation in ETL job run times until we reached 16 times our normal production workloads.”

Shrinking Footprint and Energy Costs

Not only does Fusion ioMemory system double the workload capabilities of the CRIS system, it does so on far less hardware. Mike describes how Fusion ioMemory eliminates the need for massive overprovisioning of disks: “There is a big energy awareness initiative in the US Air Force, as with other areas of the DoD. Before the refresh, this area of CRIS consisted of 5 servers and 27 arrays of disks—about a rack and a half of hardware. After adding the ioDrive cards, we reduced the system footprint to just three servers that took up about 1/16 of the space.”

Global Uptime for a Force that Never Sleeps

One might think that the three Fusion ioMemory-based servers would be fully utilized. But in fact, a single server met the USAF’s performance needs. The two additional servers delivered load balancing and redundancy that CRIS required—and without the significant maintenance overhead of the previous, disk-based system.

Mike explains “CRIS is a distributed, scale-out system with 99.97% uptime attributable to its modular architecture that allows redundant subsystems. The ioDrive cards make each piece of hardware much more powerful so it can scale farther. A single server with a Fusion ioMemory ioDrive card handles the workload as well as or better than the three servers and 21 disk arrays we used for handling active queries. CRIS uses all three servers in a distributed architecture to handle and load balance active queries, while using two of these servers in an active-passive configuration to run the ETL jobs offline.”

Summary

Implementing Fusion ioMemory gave TekSouth and the US Air Force the following benefits:

• 3X more concurrent users
• 3X more queries
• 2X greater workload
• 16:1 footprint consolidation
• 1/16th power and cooling
• Eliminated maintenance overhead for 27 disk arrays and over 400 disks

About TekSouth

TekSouth Corporation is a thirty year old, privately held firm with headquarters in Birmingham, Alabama, and strategically located program offices in the US and around the world.

Its primary services include the following:

• Performance Management, Business Intelligence and Decision Support,
• Custom Application Development, and
• IT Professional Services

TekSouth has the technical staff and management to perform large enterprise projects but remains customer-centric and able to respond rapidly to changes in requirements and technologies.