Presentation: Using AI to Optimize SQL Query Plans and Performance

Track: Sponsored Solutions Track III

Location: Liberty, 8fl.

Duration: 2:55pm - 3:45pm

Day of week:

Slides: Download Slides


Digital transformation is enabling organizations to discover valuable new ways to leverage structured and unstructured data. Queries are very powerful since they specify what information is desired from a data source. Applications issue queries to read and write tables that transform and summarize Big Data into actionable information and insights. Queries using the high level SQL language and Hive let the query engine build an execution plan, avoiding the complexity of MapReduce and promising greater efficiency and faster results with less effort.

However, cluster resources are not unlimited, and at some point the operator (and developer) will observe a query that is running more slowly than usual, or has stopped altogether. An inefficient query may pose a burden on the database’s resources, cause slow query performance, or even impact other cluster users if the query contains errors. Poor performance can result from many sources, including CPU and memory bottlenecks, an excessive number of rows, and failed stages, forcing the query operator to intervene. But without clear visibility into the query plan, resolving these issues can prove to be difficult and time-consuming.

In this informative session, Pepperdata field engineer Kirk Lewis will discuss how Pepperdata’s query performance management solution, Query Spotlight, uses AI to continuously monitor and analyze queries, providing operators with a top-down view of cluster resources combined with detailed insight into query plans, including duration time, memory/CPU usage, stages, and critical path for every query. Learn why the real-time correlation of application and infrastructure performance is essential to optimizing query plans and ensuring efficient cluster operations.

Speaker: Kirk Lewis

Sr. Field Engineer @pepperdata

Kirk joined Pepperdata in 2015. Previously, he was a Solutions Engineer at StackVelocity. Before that he was the lead technical architect for big data production platforms at American Express. Kirk has a strong background in big data.

Find Kirk Lewis at

Similar Talks

Panel: ML for Developers/SWEs


Engineering Leader @LinkedIn - AI & Big Data Enthusiast

Hien Luu

From Research to Production With PyTorch


Engineering Manager @Facebook AI

Jeff Smith

Time Predictions in Uber Eats


Leading the Machine Learning Engineering Work for Time Predictions @UberEats

Zi Wang