Presentation: Serverless Platform: Scientific Computation @Scale

Data Intensive applications are everywhere, and they present a very unique set of challenges that traditional OLTP services present. Over the last decades we have re-invented how data intensive applications work by deploying Map-Reduce at scale with Hadoop, and recently by Spark and Samza which enables stream processing thereby reducing latency for various use cases. With recent advents in Virtual Reality, Edge Computing and Deep Learning for image recognition, we are collecting more data than ever which makes it harder for existing data processing infrastructure to operate at scale when it becomes impractical to move data to compute services because of economics and limitations in resources. Further, as infrastructure is more complex, the programming model is also becoming harder and therefore makes it challenging for end users to be efficient who are doing research on new algorithmic models.

At NASA Ames Research Lab we are developing a platform for running computations as functions which would make it easier for researchers, application developers to program their applications and algorithms without any boilerplate details about the underlying infrastructure such as servers, storage shards, network, etc. This talk is going to go into the problems we are trying to solve and provide a high level overview of the platform