Presentation: Closer to the Wire: Real-time News Alerting @Bloomberg

I'm working in the News Search Infrastructure at Bloomberg. In a nutshell, when a news story hits the News Search Infrastructure, it gets enriched by a set of metadata, e.g. classification, people tagging, sentiment analysis, etc, then it is indexed to be available for searching. What is interesting is that the incoming news stories are also matched against a set of stored search queries that they users want to be alerted on in a variety of ways. My team, in particular, is responsible for the News Alerting and Streaming backend system.

News stories come into the system from a variety of different news sources. Bloomberg, itself, is one of the major sources of news, but is it’s not only a provider, but it’s also a consumer of news. As a client, you can search for new stories using a very flexible query language that we have. You can also save that search. If you save the search, you will be alerted each time a new story hits the system that matches. 90% of the time your search is translated into an alert. As a result of the live monitoring of searches, we have more alerting subscriptions than the saved alerts in the system.

What we're working on currently is how we can make the system more scalable, and how can make it more resilient. Part of this includes automated testing. That is what I'm working on at the moment.

My motivation is to share the challenges and the lessons learned from our adventure on building the News Alerting system and also show an example of a scalable alerting system. Some of the lessons learned include what you need to be aware of before deciding to go with an open source project. Software architects and tech leads will be familiar with the complex set of trade-offs that need consideration. However, it’s not rare to hear people saying “If this solution is free, then I don't have to build a team around, like I would do while building a proprietary system”. There are many benefits to using open source but in most cases of production software there is no such thing as “free lunch”.

Artificial Intelligence (AI) and Machine Learning (ML) is all the hype right now, and for good reason. Many organisations want to get ahead of the curve and use AI and ML as their competitive advantage. People used to say for most problems “Oh! There is an app for it” now we say “Oh! There is AI for it” or “There could be AI for this”. However, there have been heated discussions and news recently around the use of ML and AI, and Ethics. I feel that software development and technology is advancing so fast that we have a bad track record in stepping back and allowing time to think about an Ethics framework around them. I feel that one of the most important trend in software now and in the near future will be its alignment with Ethics.