Presentation: Online Learning & Custom Decision Services

John: There are a lot of advances in terms of interactive learning. We have a cognitive service in public preview at Microsoft now where you can go and start doing personalization of web sites very easily. You set up an RSS feed, you add a little bit javascript to your web page and then it can start personalizing based upon whatever the user prefers.

There's a sense in which it will always work once you get enough information and that makes it a much more reliable algorithm to use in practice which allows you to insert it into a lot of pipelines where you would otherwise find the overhead of hiring a data scientist and doing AB testing to be far too high.

John: There are two ways you can use this decision service. One way is for kind of pre-framed problems. Content personalization which is the one that I will be talking about is pre-framed. It's very straightforward, you sign up and create an RSS feed with the content that you want to personalize. You add a little javascript to some web pages and then it kind of happens.

On the user's browser there's a javascript call which goes out and says: “How should things be ordered?” You put that javascript call on the top of your web page. What's happening in the background is that the the structure on the web page is getting boiled down to features which are getting turned into an example which is fed into Vowpal Wabbit once joined with a click or not signal. Essentially every individual event is treated in an experiment which has a duration of 10 minutes. If a click comes in that 10 minutes it will be joined in the example and that will be something positive saying oh yes it's good, or if it's not there then it'll be negative saying it's bad.

This is for the pre-framed things where a programmer can just do it. And then there are the not pre-framed things, there you need a programmer with some data science expertise to think about how you want to solve the problem.

We have probably half a dozen different framings that we've worked on that are in different levels of development in terms of making deployable applications. The first plan is getting the kind of personalization up because we have big success stories there. The second plan is enabling things so that data scientists / Dev can do their own framings and do their own fun stuff. That's happening over the summer and the documentation for how to do your own features for example will be there by the time you run QCon. The third plan is more about trying to fully capture some framings as we develop them.

John: It's both to some extent. For the pre-framed things a developer can do it, it doesn't even have to be a back end developer, it could be just a web developer. We are creating a tool that just has not been available. For data scientists this is going to be a first kind of widely usable reinforcement based learning system.

We are reaching a point where these is a new class of problems which has never before been solvable by non PhDs via reinforcement learning and that can be a very amazing thing. Instead of taking a year of effort from somebody who has a PhD. it could be an hour of effort from somebody who knows how to program.

John: I want them to walk away knowing how to solve a new category of problems. For the ones who have those problems I want them to walk away ready to do it.

John: Machine learning is definitely very cool. Reinforcement like learning is extremely important in the long run. I expect that a decade from now we will think of machine learning as reinforcement learning because most of the problems are of that flavor.