Track: Architectures You've Always Wondered About

Location: Broadway Ballroom North, 6th fl.

Day of week:

Ever wonder how they do it? Next-gen architectures from the most admired organizations in software, such as Google, Slack, etc.

Track Host: Randy Shoup

VP Engineering @WeWork

Randy is a 25-year veteran of Silicon Valley, and has worked as a senior technology leader and executive at companies ranging from small startups, to mid-sized places, to eBay and Google. Randy is currently VP Engineering at WeWork in San Francisco. He is particularly passionate about the nexus of culture, technology, and organization.

10:35am - 11:25am

Scaling Infrastructure Engineering at Slack

In August of 2016, I was asked to build Slack’s first Infrastructure engineering organization. The company was a little over 2 years old, and we were approaching the scalability limits of the original infrastructure written by the founders several years prior. Things were starting to break in strange, and unpredictable ways.

Organizations much larger than we had initially envisioned were using Slack. Thousands of developers were building on our external APIs and stressing the system in new and unusual ways. It was taking high double digit seconds to minutes for Slack to load for very large teams, and we wanted to continue growing as fast as we could.

I’ll discuss the architectural and organizational challenges, mistakes and war stories of 2.5 years that followed, including how we:

  • Overcame the initial scalability challenges by building out our caching tier, transitioning many of our internal APIs from broadcast to publish/subscribe and rewrote many parts of our asynchronous job queueing system.
  • Continued to operate our PHP/Hack monolith, but introduced more services, and formalized how we deploy, monitor and build those services.
  • Grew the infrastructure engineering team to a global function with teams around the world.
  • Defined and cultivated an engineering-led culture in a product-led company.
  • Introduced product management, and the evolution of PM in the infrastructure team.
  • Identified key transition points when it was time to hire infrastructure specialists versus generalists.

Julia Grace, Senior Director of Infrastructure Engineering @Slack

11:50am - 12:40pm

Machine-Learned Indexes - Research from Google

Modern data processing systems are designed to be general purpose, in that they can handle a wide variety of different schemas, data types, and data distributions, and aim to provide efficient access and computation over this data. This “one-size-fits-all” nature results in systems that do not take advantage of the unique characteristics of each application, data of the user, or workload.  However, ignored in these old systems’ design: machine learning excels at understanding and adapting to particular datasets.  We present here a vision (with evidence) for the future of data processing systems: through learning models of the application, data, and workload, we can redesign and customize nearly every component of data processing systems.  We will do a deep-dive into understanding how traditional index structures can be reframed as machine learning problems, and that by doing so, and through careful model design and code synthesis, we are able to outperform cache-optimized B-Trees by up to 70% in speed while saving an order-of-magnitude in memory over several real-world data sets.  Building on these same modeling techniques, we find that we can achieve improvements in sorting, multi-dimensional indexing, and query optimization, all areas that have historically been the domain of traditional discrete algorithms and complex systems engineering.

Alex Beutel, Senior Research Scientist @Google

1:40pm - 2:30pm

Driving Technology Transformation at @WeWork

WeWork is one of the largest providers of office space in the world, opening an average of 15 new locations around the world every month. For a traditionally physical business, technology has now become an integral part of WeWork’s product offering as well as a key differentiator: software and hardware are now pervasive across the business – from managing real estate planning and construction, to building community applications, to making its buildings smarter. 

In this session, Hugo will go over the platform and architecture behind WeWork’s technology transformation over the past 2.5 years. He will outline some of the unique technology challenges WeWork faces – global systems across China and the rest of the world, hybrid infrastructure between the cloud and on-premise physical buildings, etc. – and describe in detail how WeWork is tackling them.

Hugo Haas, Fellow Engineer, Developer Platform @WeWork

2:55pm - 3:45pm

Tackling Computing Challenges @CERN

The Large Hadron Collider (LHC) at CERN is the world's most powerful particle accelerator and is one of the largest and most complicated machines ever built. The LHC has been vital in helping physicists make new discoveries such as the Higgs boson in 2012. Today, the Worldwide LHC Computing Grid regularly operates a million processor cores and nearly an exabyte of disk storage. By 2026, the successor to LHC will require 50-100 times more computing capacity and will store multiple exabytes per year.  

In this talk, I will discuss the current challenges of capturing, storing, and processing the large volumes of data generated by the LHC experiments. I will also discuss our ongoing research program at CERN openlab to explore alternative approaches, including the use of commercial clouds as well as alternative computing architectures, advanced data analytics, and deep learning.  

You will learn how a highly data-intensive organization can effectively take advantage of future computing improvements in both hardware and software. You will also gain some interesting insights into modern high energy physics.

Maria Girone, CTO @CERNopenlab

4:10pm - 5:00pm

Video Streaming at Scale

Let's face it - we all want to be there when the action happens, not experience it after the fact. That's why live video streaming has become so popular. But what happens when you want to reach millions of people across the globe in real-time. At scale, there's a lot going on.
In this talk, Lysa Banks, IBM Distinguished Engineer, will take us "behind the scenes" and deep dive into how live video streaming happens at scale. Lysa will cover the core technology stack and application architecture for IBM Streaming Manager (formerly Ustream) used by individuals and enterprises to reach massive audiences for everything from gaming to product launches to major conferences. This talk will not only cover the current architecture but also how the platform is evolving to be event-driven and microservices centric. It will also outline improvements that were made in processes in order to grow from startup to enterprise-class operations.
This talk will cover these specific topics:
  • How live streaming is done in the cloud (Architectural Overview)
  • A philosophy and framework for decoupling monolithic applications to microservices
  • Moving from the data center to cloud VMs to Kubernetes
  • How microservices enable efficient scaling in the cloud
  • How multicloud is leveraged for increased reliability 
  • Shifting the team's mindset from "homegrown and roll your own" to open source and SaaS for increased focus on core competencies
  • Changes made to security and DevOps practices to better serve enterprise clients

Lysa Banks, IBM Distinguished Engineer, CTO Watson Media Cognitive Solutions @IBM

5:25pm - 6:15pm

Architectures Open Space

Presentation details to follow.


This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.