Presentation: Serverless Platform: Scientific Computation @Scale

Track: Immutable Infrastructures: Orchestration, Serverless, and More

Location: Broadway Ballroom South, 6th fl.

Day of week:

Slides: Download Slides

Level: Intermediate - Advanced

Persona: Architect, Developer, DevOps Engineer

Abstract

Data Intensive applications are everywhere, and they present a very unique set of challenges that traditional OLTP services present. Over the last decades we have re-invented how data intensive applications work by deploying Map-Reduce at scale with Hadoop, and recently by Spark and Samza which enables stream processing thereby reducing latency for various use cases. With recent advents in Virtual Reality, Edge Computing and Deep Learning for image recognition, we are collecting more data than ever which makes it harder for existing data processing infrastructure to operate at scale when it becomes impractical to move data to compute services because of economics and limitations in resources. Further, as infrastructure is more complex, the programming model is also becoming harder and therefore makes it challenging for end users to be efficient who are doing research on new algorithmic models.

At NASA Ames Research Lab we are developing a platform for running computations as functions which would make it easier for researchers, application developers to program their applications and algorithms without any boilerplate details about the underlying infrastructure such as servers, storage shards, network, etc. This talk is going to go into the problems we are trying to solve and provide a high level overview of the platform

Speaker: Diptanu Choudhury

Cloud Infrastructure Engineer @Facebook

Diptanu is an infrastructure engineer, and works on large-scale distributed systems, cluster schedulers, service discovery and highly available and high throughput systems on the public cloud. He is an author of the upcoming Oreilly book on Platform Engineering. He is a core committer to the Nomad cluster scheduler which has a parallel and distributed scheduler and support heterogeneous virtualized workloads. Prior to HashiCorp, Diptanu worked in the Cloud Platform group at Netflix, where he worked on the core platform infrastructure that powered the Microservices infrastructure of Netflix. He worked on Apache Mesos and wrote a cluster scheduler for running clusters of Docker containers on AWS, and also contributed to various reactive IPC and service discovery infrastructure projects.

Find Diptanu Choudhury at

Similar Talks

Let's talk locks!

Qcon

Software Engineer @Samsara

Kavya Joshi

Graceful Degradation as a Feature

Qcon

Director of Product @GremlinInc

Lorne Kligerman

Liberating Structures @CapitalOne

Qcon

Agile Coach, Engineering @CapitalOne

Greg Myers

Making 'npm install' Safe

Qcon

Software Engineer @agoric

Kate Sills

Driving Technology Transformation at @WeWork

Qcon

Fellow Engineer, Developer Platform @WeWork

Hugo Haas