Many database researchers in recent times have toyed with the idea of building a “learning optimizer” that grows smarter and more efficient over repeated use. There have been many recent advances in applying machine learning to data management challenges. Most of these solutions typically solve well-defined subtasks of the query optimizer, such as cardinality estimation or join ordering, though some attempt to solve the entire problem of query optimization end-to-end. These approaches tend to construct a neural network that takes as input a partial query plan and predicts the best expected performance that the system could produce if this plan were to be completed. The neural network iteratively learns the best ways to complete plans based on its past performance. These techniques are, to the best of my knowledge, largely unexplored in the data federation space and I think Presto is the right vehicle for the experiment. Exciting times ahead!
Vivek Bharathan is cofounder and principal software engineer at Ahana. Extremely passionate about big data, distributed systems, and analytic databases, Vivek has more than 10 years of experience as a core database engineer. Most recently, Vivek was a software engineer at Uber where he managed Presto clusters with more than 2,500 nodes, processing 35PB of data per day, and where he worked on extending Presto to support Uber’s interactive analytics needs. His Presto contributions include the pushdown of partial aggregations. Vivek holds an M.S. in computer science and engineering from Ohio State University.
—
New Tech Forum provides a venue to explore and discuss emerging enterprise technology in unprecedented depth and breadth. The selection is subjective, based on our pick of the technologies we believe to be important and of greatest interest to InfoWorld readers. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content. Send all inquiries to newtechforum@infoworld.com.