Track: Architectures You've Always Wondered About

10:35am - 11:25am

Scaling Infrastructure Engineering at Slack

In August of 2016, I was asked to build Slack’s first Infrastructure engineering organization. The company was a little over 2 years old, and we were approaching the scalability limits of the original infrastructure written by the founders several years prior. Things were starting to break in strange, and unpredictable ways.

Organizations much larger than we had initially envisioned were using Slack. Thousands of developers were building on our external APIs and stressing the system in new and unusual ways. It was taking high double digit seconds to minutes for Slack to load for very large teams, and we wanted to continue growing as fast as we could.

I’ll discuss the architectural and organizational challenges, mistakes and war stories of 2.5 years that followed, including how we:

• Overcame the initial scalability challenges by building out our caching tier, transitioning many of our internal APIs from broadcast to publish/subscribe and rewrote many parts of our asynchronous job queueing system.
• Continued to operate our PHP/Hack monolith, but introduced more services, and formalized how we deploy, monitor and build those services.
• Grew the infrastructure engineering team to a global function with teams around the world.
• Defined and cultivated an engineering-led culture in a product-led company.
• Introduced product management, and the evolution of PM in the infrastructure team.
• Identified key transition points when it was time to hire infrastructure specialists versus generalists.

11:50am - 12:40pm

Machine-Learned Indexes - Research from Google

Modern data processing systems are designed to be general purpose, in that they can handle a wide variety of different schemas, data types, and data distributions, and aim to provide efficient access and computation over this data. This “one-size-fits-all” nature results in systems that do not take advantage of the unique characteristics of each application, data of the user, or workload.  However, ignored in these old systems’ design: machine learning excels at understanding and adapting to particular datasets.  We present here a vision (with evidence) for the future of data processing systems: through learning models of the application, data, and workload, we can redesign and customize nearly every component of data processing systems.  We will do a deep-dive into understanding how traditional index structures can be reframed as machine learning problems, and that by doing so, and through careful model design and code synthesis, we are able to outperform cache-optimized B-Trees by up to 70% in speed while saving an order-of-magnitude in memory over several real-world data sets.  Building on these same modeling techniques, we find that we can achieve improvements in sorting, multi-dimensional indexing, and query optimization, all areas that have historically been the domain of traditional discrete algorithms and complex systems engineering.

1:40pm - 2:30pm

Driving Technology Transformation at @WeWork

WeWork is one of the largest providers of office space in the world, opening an average of 15 new locations around the world every month. For a traditionally physical business, technology has now become an integral part of WeWork’s product offering as well as a key differentiator: software and hardware are now pervasive across the business – from managing real estate planning and construction, to building community applications, to making its buildings smarter. 

In this session, Hugo will go over the platform and architecture behind WeWork’s technology transformation over the past 2.5 years. He will outline some of the unique technology challenges WeWork faces – global systems across China and the rest of the world, hybrid infrastructure between the cloud and on-premise physical buildings, etc. – and describe in detail how WeWork is tackling them.

2:55pm - 3:45pm

Tackling Computing Challenges @CERN

The Large Hadron Collider (LHC) at CERN is the world's most powerful particle accelerator and is one of the largest and most complicated machines ever built. The LHC has been vital in helping physicists make new discoveries such as the Higgs boson in 2012. Today, the Worldwide LHC Computing Grid regularly operates a million processor cores and nearly an exabyte of disk storage. By 2026, the successor to LHC will require 50-100 times more computing capacity and will store multiple exabytes per year.  

In this talk, I will discuss the current challenges of capturing, storing, and processing the large volumes of data generated by the LHC experiments. I will also discuss our ongoing research program at CERN openlab to explore alternative approaches, including the use of commercial clouds as well as alternative computing architectures, advanced data analytics, and deep learning.  

You will learn how a highly data-intensive organization can effectively take advantage of future computing improvements in both hardware and software. You will also gain some interesting insights into modern high energy physics.

4:10pm - 5:00pm

Video Streaming at Scale

5:25pm - 6:15pm

Architectures Open Space

Presentation details to follow.