Presentation: Control Planes: Designing Infrastructure for Rapid Iteration

I've been at Clever for since the start (five and a half years while the company itself started around six years ago).  I joined as a software engineer, started focussed on infrastructure and security and also started the infrastructure team. Due to my tenure, I do end up dealing with things such as old database instances and the alerts and metrics for them.

However, as a technical product manager for infrastructures and security most of my time is spent in planning for the coming year. What that means is, for now, our biggest focus is on resiliency. We are expecting to grow some of our core usage like 4 to 5x within a month.

Clever is an education company, so we get all of our users in one month between August and  September. We drop down to very low usage during the summer as everybody's on vacation and then when people come back, all of the work that we've done over the last year sees use. And even that happens in chunks when the East Coast wakes up and kids go to school for example.

We are an enterprise company, but for the education space. So, we connect school districts, primarily public school districts, to education apps that they use in the classroom. We provide everything from account management, security (and thinking of data), single sign-on and a portal.

If you use Chromebooks which is most schools in the country are using right now, you log in to the Chromebook using Clever. Kids can log into the Chromebook without entering a password by using Clever Badges (which use QR codes).

Basically, when a class starts, we get thousands of authorization requests of different kinds from different schools.

Yes. Using microservices was not an explicit decision that we ever made. We just found ourselves there. So, you know, we have been on about 400 different applications in our cluster, and we have 40 engineers.

Yes, but we started on Node.js and MongoDB. MongoDB is still our primary data store, but we have Polyglot database store right now and most of our backend services are in Go.

We are an education-focused company that works with public schools. Most employees at Clever, joined the company to make impact in the classroom. Engineers at Clever really care about product delivery. They do care about solving complex problems, but they mostly care about the customer.

This is the primary driving factor for our technical infrastructure. How can we as an infrastructure team drive ourselves out of business every six months? We are a small team and only want to be solving problems that directly affect our customers. When we saw Docker, we realized that would allow engineers to focus very clearly on their application and completely isolate themselves from ‘infrastructure needs’.

Early on use of Docker suddenly took us over, just because everybody wanted to use it, and we were waiting for it. We rewrote scripts so the Docker containers would go on standard EC2 instances. It was just running one Docker container on an instance or two Docker containers on an instance, with no orchestration. But it pretended to be an orchestration system.

Coming to your question, the story that I think that is exciting is how, like most things, we drove our decision making based on what engineers wanted to use, building the user interface and the tooling and then using smoke and mirrors in the background to make that happen.

That allowed us to look at what our needs were at a specific time. For example, the first issue that we faced was we were developing a lot of new asynchronous jobs and we had to deploy them quickly. EC2 instances were becoming slow and becoming too expensive. So we had to make a change. Mesos was a system that we used then because we couldn't figure out a good solution of getting the load balancing to work or services to work right and we had to do asynchronous work.

We kind of moved to Mesos and then we had a couple of senior engineers look into how you get load balancing working. While we were doing that, Kubernetes become big, so we started looking into it. We used Kubernetes for services for a little bit, and then ECS came out which allowed us to kind of use our existing infrastructure and move much faster than we were with speed of Kubernetes at that time.

The main audience is architects, engineers, and infrastructure engineers. Anybody who cares about velocity or resiliency of an engineering team. I think that is the real focus  is around organizational and team level engineering productivity.

I’d like engineers, through our stories to be able to create space for technical experimentation while building complex systems. And to evolve rapidly without thrashing the engineering team’s velocity. We focus a lot of allowing us time to think about decisions carefully while still also delivering tools to engineers.

Even with infrastructure, you can start from the user interface. You don't have to solve all the technical problems first. You can look further into the future to your ideal architecture, knowing that others will fix many of those problems because many others are facing the same issues. Some features are more important for your team than everybody else. Those are the solutions you need to be focussed on.