Presentation: Serverless Sec & Things That Go Bump in the Night

Erik: I’ve been in software and IT Security for almost 2 decades now, with almost 9 years architecting cloud solutions. In 2009 I joined Veracode to help them build new cloud and app security products as part of the CTO’s office. That all changed recently though when in August 2016 I left to found CloudZero and focus on Cloud development, operations and security challenges 100%. Since then I'm a CEO, engineer and chief coffee maker.

Erik: I started working with Amazon almost nine years ago. Along this path you go through all the different phases of cloud adoption as you explore the technology and the platform itself evolves. You start out thinking about virtualization, then you discover services like RDS and stop building your own database and then you keep going. Ultimately you stop building your applications and start assembling them and this is when you start to ask "Why am I even using servers anymore?”

So you start to ask “what about this serverless thing and AWS Lambda?” At that point, you have really opened pandora's box and it’s easy to get confused. Lots of different points of view right now which is typical of a early technology. For example, there is a lot of groaning on even the term “Serverless” where some are asking if the right term is really Functions as a Service (FaaS)? The reality is that neither serverless or FaaS is really the right way to think about it. The way I see it, and this is something I've been saying for quite a long time, is that we should all be thinking of the Cloud as a new Operating System and the thing that's been missing is the native code of this operating system and that’s Serverless.

For example, I got back from the serverless conference about a month ago, and everybody was trying to define it. It’s ok, it’s a fairly typical reaction to a new technology that is just emerging as people get excited about it. I think in a few years time though we won’t be talking so much about what “is” Serverless but more about the entire “Cloud Operating system” and how things like AWS Lambda are just one part of this cloud native system. Essentially serverless or rather this whole system is event-driven computing on invisible infrastructure. That important piece about invisible infrastructure is really the key takeaway. It absolves you from having to build a lot of these things, but keep in mind it doesn't absolve you however of being responsible for the end result. You can outsource the work but you can't outsource the responsibility. And, that's essentially what you're doing, outsourcing a lot of the work that you have to deal with as a developer building applications, so you can create amazing things. Out of this you find yourself in a world where it's really hard to build something that just doesn't scale.

As an architect you probably spend the time trying to figure out how to survive a runaway success moment. With serverless at first it feels like it's hard to build an application that doesn't necessarily scale, until you actually start doing things at scale and all of these dangerous emergent properties of the system emerge. And on top of this how do you add security? Today security should be just one aspect of architecting a well-constructed application, how well will you survive in this hostile environment, the cloud's operating system. In this event-based infrastructure we have to think about things that we didn't necessarily think about before. Some things are old problems that we now have to be more concerned about, such as code security, Can people inject attacks, do I have known vulnerabilities in my code? Things like cross-site scripting or SQL injection don't necessarily go away, they just have different ramifications and impact.

I the past I've given a talk about the cloud full stack hack. If you build any sort of complicated system these emergent properties that come out of the computer environment that coalesce and create vulnerabilities and problems that you couldn't imagine before. It's not one individual thing that kills you. It's a combination of many little things that come together and conspire against you, and the impact is not just that somebody necessarily gets access to your customers data. That's pretty bad. The impact is that your attacker has access to your entire cloud datacenter, and they can take your customer data without a trace and then delete everything, all traces of your system from ever existing on the Internet.

As people start doing serverless, they tend to forget that they're creating code that is receiving in many cases the full brunt of the Internet, without the typical layers we try and put in front of our applications. Amazon or Google is not going to solve these problems, you still have to build security in but you have to think about new things, such as "Who can invoke my functions or trigger events?" It's not just about network access controls and what ports are open. It's about permissions, it's about identity access management policies, it's about how all services are woven together, and all the permissions that I use that are necessary to make that work. And one of the problem with permissions is that these things tend to grow, and it's very difficult to shrink them. So how do I practice the least privilege? Who can invoke my functions is suddenly an interesting problem. It's not just a network layer, it's an identity and access management issue. What can my functions do once they're executing? What services do they have access to? Do they have access to too much? Do they have access to the code itself? Could my function give up its source code to an attacker and give them the ability to explore that?

Then, if my function is out there and anybody can can hit it, or if it's part of an application or a conglomerate of functions working together that is triggered by some user behavior, I have a new problem which is called "denial of wallet" where my attacker doesn't necessarily attack me from a security point of view, nor from a denial service point of view (because everything now scales so well!), but from a billing point of view. They can hit my function over and over again, and because Lambda scales so well it's all too happy to process terabytes of worthless data. That worthless data I still have to pay for at the end of the month. Some will say, I'll limit that, and then next thing you realize is now you have just made it possible to have a denial of service problem.

You have to think about the flow of events, data, activities and your application in a way that you probably haven't thought about. You can easily evaluate the cost of an EC2 instance or of a server, but it is not as clear and transparent to use Lambda functions by the millisecond. Infinite scale also means infinite cost.

The other question to ask is just what is my application's attack surface? It’s one thing to say there's my application, it's packaged up in a jar file, or it's Python package, or a zip file. I've deployed it. It's running over there, or it's on this server, or it's on this cluster of Docker containers. But when I create a fairly complex serverless application which is made up of dozens or more of service functions, what exactly is the surface of my application which means what is the attack surface of my application, and how do I secure that? How do I go about making sure that I know how it can be accessed? And if each function is done by an individual developer, how do we make sure that they all are coordinating, to make sure that they they're not opening up some back door that they don't necessarily know about.

The challenge here is that people can see serverless as glue code used to bring together lots of different services. The danger of glue code is that you might not treat that code as a first class citizen. It means it doesn't go through the ordinary process. It doesn't go through the same testing regimen. You don't necessarily follow the same disciplines, and it encourages a lot of bad habits. As a seasoned developer you might find yourself making those mistakes because you may say, I can solve that problem in two seconds by going to the Amazon console, typing in some code, and it just works. How am I going to test that? How am I going to secure that? How am I going to ensure that I've understood the impact my change has on my entire infrastructure? All this flexibility involves a huge risk.

Erik: It is a question that I will address in my talk and one that we all should be asking ourselves as we're building up any application. And it is not an easy question to answer because serverless has such an amazing complexity and such a poorly developed tool chain. And it's nobody's fault. It's because it's very fast moving. And there are no tools to map out the attack surface so, but if you have taken a more declarative approach to creating applications you have some notion of what your surface is defined in those declarations used to set up your services. But the other piece is that this architecture is dynamic. You've got your serverless functions calling out to other functions which might be calling out to other platform services environments and then they start calling other things, so you get this cascading effect.

You have to map that out. And if you're working on a team, you have these trust boundaries that you have to maintain between all your functions and the services that you're calling. And it's a huge challenge for just two developers working on the same project but managing different functions between the two of them to make sure that they're establishing the least privilege between them. That failure to configure the privileges correctly leads to attack surface leaks all over the place, and that gives the attacker the ability to start poking and prodding at these things.