Data management paper

Hadoop clusters, often outside of normal
business hours so as to avoid impacting the
performance of the database. Enterprises
that employ this data load model are limited
to analyzing data that can be hours or days
old, which diminishes the ability to act on
quickly changing trends.
Terracotta BigMemory Max* Enables
Real-time Big Data Analysis
A combination of factors—including
advances in CPU and networking
technologies, and the creation of advanced
data gathering and analysis tools such
as Apache Hadoop and the MapReduce
framework—changed the analysis
landscape forever. Now, enterprises are
pushing the envelope even further with
in-memory data management and analysis.
Terracotta provides a high-performance
alternative to disk-based storage with
Terracotta BigMemory Max. BigMemory
Max overcomes the weaknesses of
traditional batch-loaded data analysis
by storing data in the server’s machine
memory. Enterprises can use this data
storage space to store terabytes of
data in machine memory, which is orders
of magnitude faster than loading data
onto disk-based storage. Storing data
in memory provides low, predictable
latency and eliminates delays associated
with disk access. These features make
data available instantly for analysis, with
BigMemory Max automatically optimizing
memory use with changing data patterns.
BigMemory Max can use as much memory
as is physically available on the server,
and provides the option of building arrays
of interconnected servers to provide an
even larger memory storage footprint.
With BigMemory Max, enterprises can
reduce the amount of time required
for complex analysis from hours to
minutes, or even seconds, which opens up
unprecedented opportunities to quickly
turn high-velocity transactional data into
actionable business decisions.
Terracotta is already a recognized leader
of in-memory data management solutions.
CERN, the European Organization for
Nuclear Research, is one of the world’s
leading particle physics research centers.
The organization provides scientists
from around the world with the ability to
conduct high-energy physics experiments
using equipment such as the Large Hadron
Collider (LHC) particle accelerator. CERN
uses a variety of control systems to
monitor critical equipment, including the
Technical Infrastructure Monitoring (TIM)
platform, which monitors approximately
120,000 sensors, and Diagnostics and
Monitoring (DIAMON), which monitors
some of the equipment related to the
particle accelerators.
TIM and DIAMON require a high level
of performance and availability. CERN
engineers used Terracotta BigMemory
Max to build a stable, redundant system
that can process incoming data in less than
one second. As a component of the CERN
Control and Monitoring Platform (C2MON),
Terracotta BigMemory Max holds current
status data from TIM and DIAMON in main
memory, and provides high availability by
supporting automatic failover between
mirrored servers. The combination of in-
memory data storage and high availability
improves overall system response times,
increases throughput, and helps C2MON
achieve 99.99984 percent availability for
the TIM platform.
1
The advantages of BigMemory Max also
extend to the nancial services industry. A
Fortune 500* online payments processor
relied upon a platform that could only
handle 50 fraud detection rules, and
was limited to 2,000 fraud detection
transactions per second. Latency was also
inconsistent. As a result, the company failed
to meet service-level agreements (SLAs)
as often as ten percent of the time, costing
an estimated $10 million a year in nes and
fraudulent charges. To help reduce SLA
violations and fraud and improve customer
satisfaction, the company built a high
performance system with BigMemory Max
that can hold over 100 times the number
of fraud detection rules in memory, boost
transaction throughput by over 30 times
to 65,000 transactions per second, and
allow the company to approve or ag
transactions within 650 milliseconds. The
company estimates it will save tens of
millions of dollars by avoiding missed SLAs
nes and fraudulent charges.
2
Table of Contents
What’s Holding Back Real-time
Big Data Analysis? ............... 1
Terracotta BigMemory Max* Enables
Real-time Big Data Analysis ....... 2
How BigMemory Max Works ....... 3
How Applications Access
BigMemory Max ................ 3
Terracotta and Intel: Powering the
New Generation of Data Analysis .. 3
Scalability ..................... 3
Performance .................. 4
Reliability ..................... 4
Lower Operating Costs ......... 4
The Performance Advantage of
Intel Technologies. . . . . . . . . . . . . . . . 4
Conclusion ...................... 6
Terracotta and Intel: Breaking Down Barriers to In-memory Big Data Management
2
Terracotta and Intel: Breaking Down Barriers to In-memory Big Data Management