It isn’t easy to visualize an expansive body of water living within a mass of water vapor in the sky, and understanding cloud data lakes can be just as difficult. In an effort to educate and illuminate IT professionals, Dremio presented the first Subsurface Digital Conferenceon July 30, 2020 –a free online event dedicated to exploring the future of cloud data lakes. The conference brought together industry experts from a wide variety of organizations with valuable experience across the vast tech ecosystem. Panelists included representatives from open source and technology leaders such as Netflix, Expedia, Exelon, Microsoft, and AWS,among others, to explore “what lies beneath the surface of data lakes.”
Cloud data lakes allow organizations to better analyze and manage their data by separating compute and storage, providing increased scalability, better availability, and potentially lower costs. Effective data lakes also minimize data lock-in, making it as easy to export data as it is to import. Dremio CEO Billy Bosworth led the conference with an opening keynote titled “The Way Forward,” suggestingthatthose involved look beyondthe hype around data lakes and focus on improving access, capability, and agility of the data that the architectures provide. His colleague Tomer Shiran, Dremio’s co-founder and CPO, also spoke on “The Rise of the Cloud Data Lakes,” chronicling the evolution beyond monolithic architectures and the benefits of separating compute from data. Shiran envisions a future of data built on open source and open standards, circumventing vendor lock-in and data silos.
Netflix data engineer Daniel Weeks brought to light the streaming juggernaut’s reliance on data in every aspect of its operations. Weeks, who leads the company’s Big Data Compute Team, recounted the creation of Apache Iceberg, an open table format for massive analytic datasets that began as an internal project. Though it is still in development, Iceberg promises to be a major utility in the future of Big Data. Another notable keynote was delivered by Tomasz Tunguz, managing director of Redpoint Ventures, counting down “Five Data Trends You Should Know.” The Dremio board member detailed the rise of the data engineer and the importance oftrends like data pipelines, compute engines, data modeling, and more. Dremio, a premier data lake engine vendor, is planning a second conference in January 2021.
