Another time-saving trick: The software keeps a metadata table that lists the high and low key values for each data page, or column of data. So when a query is executed, the database can check to see if any of the sought values are on the data page."If the page is not in memory, we don't have to read it into memory. If it is in memory, we don't have to bring it through the bus to the CPU and burn CPU cycles analyzing all the values on the page," Vincent said. "That allows us to be much more efficient on our CPU utilization and bandwidth."With columnar processing, a query can pull in just the selected columns of a database table, rather than all the rows, which would consume more memory. "We've come up with an algorithm that is very efficient in determining which columns and which ranges of columns you'd want to cache in memory," Vincent said.
On the hardware side, the software comes with parallel vector processing capabilities, a way of issuing a single instruction to multiple processors using the SIMD (single instruction, multiple data) instruction set available on Intel and PowerPC chips. The software can then run a single query against as many columns as the system can place on a register. "The register is the most efficient memory utilization aspect of the system," Vincent said.
IBM is not alone in investigating new ways of cramming large databases into the server memory. Last week, Microsoft announced that its SQL Server 2014 would also come with a number of techniques, collectively called Hekaton, to maximize the use of working memory, as well as a columnar processing technique borrowed from Excel's PowerPivot technology.
Database analyst Curt Monash, of Monash Research, has noted that with IBM's DB2 10.5 release, Oracle now is "now the only major relational DBMS vendor left without a true columnar story."
IBM itself is using the BLU components of DB2 10.5 as a cornerstone for its DB2 SmartCloud infrastructure-as-a-service (IaaS), to add computational heft for data reporting and analysis jobs. It may also insert the BLU technologies into other IBM data store and analysis products, such as Informix.