Adding speed and scalability to existing applications with in-memory data grids

Powering data-intensive and time-sensitive applications with in-memory data grids (IMDGs)

blazing speed leaving a glowing trail along a road

In the era of digital transformation, omnichannel marketing, web-scale applications, and the internet of things (IoT), cost-effectively scaling the performance of existing applications is one of the most challenging issues facing enterprise architects and CTOs. In-memory data grids (IMDGs) meet this challenge, delivering massive speed and scalability gains without the need to rip and replace existing applications or data layers.

Until recently, the most common option for massively scaling applications was to purchase additional expensive hardware—over and over again. This severely limited the ability of most companies to aggressively pursue the major initiatives, such as digital transformation, that would most effectively increase their competitiveness. At best, the return on investment of these expensive projects was low.

Today, IMDGs offer a simple, cost-effective alternative. An IMDG consists of a cluster of servers that shares the available memory and CPU power, evenly distributes the dataset across the cluster nodes, parallel processes compute on the node where the applicable data resides, and allows scaling simply by adding a new node to the cluster. Inserted between the application and data layers, the IMDG moves a copy of the disk-based data from RDBMS, NoSQL, or Hadoop databases into RAM. This allows processing to take place without the delays caused by continually having to read and write the data from disk.

In a public or private cloud environment, nodes can be added or subtracted from the IMDG cluster as-needed for maximum flexibility and cost-effective scaling. Some IMDGs also support ANSI-99 SQL and full ACID transactions, advanced security, stream processing support, machine learning, and Spark and Hadoop acceleration.

IMDGs in action

The IMDG is a well-established and continually evolving technology. It is being used by well-known brands for their most mission-critical applications.

Workday, the popular leader in cloud solutions for financials and HR, is using its IMDG to support approximately 1,800 enterprise customers, which includes a third of the Fortune 500 and a third of the Fortune 50. With approximately 26 million workers under management, Workday uses its IMDG to process about 189 million transactions per day, with a peak of about 289 million per day.

Wellington Management is using its IMDG to power its investment book of record (IBOR), which is the single source of truth for investor positions, exposure, valuations and performance. All real-time trading transactions, all related account activity, and all related back-office activity flow through the IBOR in near real time, which also provides analytics to support performance analysis, risk assessments, regulatory compliance and more on a hybrid transactional/analytical processing (HTAP) architecture.

FSB Technology (UK) Ltd. provides real-time sports betting platforms as a fully managed service. Thanks to its IMDG, FSB is supporting more than 500 casino and live dealer games, processing more than 700 bets per second, and transacting over £2 million in bets daily. The IMDG also makes it possible for FSB to make huge amounts of constantly updated event data immediately available to a vast number of clients and to deploy public cloud nodes to complement its on-premises nodes during peak traffic periods.

Sberbank, Russia’s largest bank, is building an IMDG to power its 24/7 online and mobile banking infrastructure. The system will store and process 1.5 petabytes of data in real time and support thousands of transactions per second by 135 million customers. The 2,000-node data grid will be on par with the largest supercomputers in the world, providing more data storage capability with a little less compute power.

Selecting an IMDG

Before choosing an IMDG, consider the following criteria to make sure you are adopting the right technology for your needs. You likely need an IMDG if:

  • The performance of your existing applications cannot support increasing user volume with acceptable performance.
  • Real-time responsiveness is required.
  • The application would benefit from accessing data from a distributed, highly available data layer.
  • You lack the time or budget to lift and shift the application to an in-memory database.
  • You would benefit from simplifying your overall system architecture by moving to HTAP.
  • You need a flexible, scalable architecture that can be deployed on-premises, in the cloud, or in a hybrid environment.
  • Your plan is to shift the application architecture to an in-memory database over time and the IMDG, as part of an in-memory computing platform, allows you to first move painlessly to in-memory computing, and then eventually easily shift to an in-memory database on the same platform.
  • Your organization would benefit from standardizing on a single in-memory computing platform to reduce the number of technologies in your infrastructure.

When evaluating IMDGs, look only at in-memory computing platforms that include an IMDG and an in-memory database to ensure an easy migration in the future. Also consider the following to ensure the data grid will support your needs for the long term:

  • ANSI-99 SQL support, so the IMDG can be easily inserted into your existing SQL-based architecture.
  • ACID transaction support, so the platform can be used as the system of record for valuable applications in production environments.
  • Platform and language flexibility to ensure ease of integration and maximum ROI.
  • Machine learning support to empower current or future in-process decision making.

When looking at technologies to power your most data-intensive and time-sensitive applications related to digital transformation, omnichannel marketing, web-scale applications and the internet of things (IoT), be sure to explore the power of in-memory data grids to cost effectively—and massively—speed up and add scalability to your existing applications.

This article is published as part of the IDG Contributor Network. Want to Join?