Presentation: Nonconformist Resilience: DB-Backed Job Queues

I lead software architecture at Betterment, which means I work with people throughout Betterment’s engineering team, keeping apprised of new developments and challenges throughout the org, sharing and cross-pollinating best practices and a shared vision for our platform, and regularly diving deep into the code alongside domain owners

John: A lot of companies end up selecting patterns based on industry norms, but sometimes the accepted patterns have rough edges that may permanently leak into your app layer causing pain. There’s a strong sense in the industry currently that you should never use a database as a job queue, instead delegating to a product that’s called a queue. And there are valid reasons to prefer a dedicated queue, but there are also reasons not to, which often get short shrift. At Betterment, we build a suite of products that people rightly care a great deal about the correctness and consistency of, and folks don’t generally realize that when you coordinate across two datastores (which a queue is) how hard a problem it is to perform a transaction that also enqueues background work, and then ensure that that background work definitely gets worked if-and-only-if the transaction commits. Many folks will end up addressing the edge cases in their business logic on a per-feature basis rather than simply eliminating the problem by unfashionably using the database as a work queue.

There will definitely be pushback from some folks on the basis of scalability and throughput - and those are real concerns for some applications, but certainly not all, and in many cases, there are other levers you should be thinking about pulling to alleviate those concerns rather than switching jobs to a dedicated queue. I’ll be presenting an honest warts-and-all accounting of the tradeoffs so that ideally folks in the room can apply them to their distinctive problem spaces and come away with better outcomes, fully aware of the pros and cons of the choices they make.

John: Engineers building new platforms or evaluating technology for future revs of their platforms would be the sweet spot. Background work is something that most grown-up apps need to perform, and there doesn’t seem to be much info out there about the pros and cons of different approaches.