QCon New York June 15-19, 2020 | | Git Gud with Property-Based Testing

This presentation is now available to view on InfoQ.com

What You’ll Learn

Learn how property-based testing can improve the quality of your code base.
Hear how Polysync is using property-based testing in place of integration tests.
Understand that capabilities and limitation of property-based testing.

Abstract

Traditional unit tests force developers to make difficult design decisions on their own, meaning their tests can be only as robust as the human imagination allows. In searching for an alternative to this limiting style of testing, I’ve found Property-Based Testing as a meaningful alternative. Property-Based Testing presents a marked shift from this typical awkward style of test creation. Instead, test cases are programmatically generated to try to explore not only execution path coverage but also the domain input space. This bolsters the possibility of exposing unidentified problematic inputs and states.

Typically, Property-Based Testing is applied close to the code in order to exercise functions. I’ve found that it can also be meaningfully used for testing at the integration level. Over the course of my discussion I will cover my experience using both methods as a way to find casting and overflow bugs in PolySync’s Open Source Car Control (OSCC) project, as well as for work involving the validation of security research and hardening strategies for git repositories.

Question:

What is the focus of your work?

Answer:

Today, I am working on using property-based testing to create various different kinds of distributed git repository scenarios. The goal is to validate that our solution to the git metadata vulnerability is working as designed. So I am trying to create any kind of wacky system state to ensure that our solution still works.

Question:

You work for a Self-Driving-Car System Company. You are talking about git and you are talking about property-based testing. What’s the connection?

Answer:

We are trying to secure our deployment tools so we can ensure that nothing is being tampered with. This git test work is based on a security vulnerability that was exposed in a paper (On Omitting Commits and Committing Omissions: Preventing Git Metadata Tampering That (Re)introduces Software Vulnerabilities). We are trying to make sure our repositories are secure, and we are delivering what we are saying we are delivering in a way that does not mess up the data.

Question:

What do you hope people get out of the talk?

Answer:

I’d like to let people know that property-based testing is a really great way to enhance your testing While property-based testing seems like a scary concept to get into, if you can learn how to correctly use it your tests will be more robust and more interesting to develop. It also provides more test cases for less work than traditional testing methods.

Question:

Is this a code talk or a case study talk, are you showing people how to write a test or how to implement a program?

Answer:

It is pretty much just going to be based on my experiences learning property-based testing. When I started testing, I really was not familiar with testing much at all. I did some unit testing and cucumber style integration tests and then very quickly switched over to property based tests. Making that shift from traditional testing to property-based testing benefited us because we found certain bugs that we would not have found out with regular unit testing.

Question:

Can you give me an example of one of the stories you will tell in the talk?

Answer:

When I first started using property-based testing, it was on OSCC (or Open Source Car Control platform). OSCC is a lot of Arduino firmware and stuff like that. We had multiple unit tests written for it but we were pretty much considering typical usage. With the property based tests, we found overflow errors or casting errors where we were storing to Int32s. For example, if you passed a negative number it would underflow. Since we were not considering these conditions, it pretty much resulted in an overhaul of a bunch of that code when we realized that a lot of the internal stuff was not safe as we thought it was.

Question:

How much of the talk is about git in this metadata vulnerability versus property-based testing? Is it about both or is it about property-based testing?

Answer:

It is about both. To introduce attendees to the benefits of property-based testing, I will be using examples from our work both on supplementing our unit tests for OSCC and creating system tests for the git metadata solution.

Question:

What do you feel is the most important trend in software right now?

Answer:

I think the most important trend is going to be shifting away from the importance of AI and algorithms and shifting to a focus on safety - making sure everything works correctly because it’s important to feel confident that software is correct before putting it into a vehicle and out on the road. What I am trying to say is I think automation is really important. I think it is going to make a lot of things safer and a lot of things easier.

Speaker: Katie Cleary

Software Engineer @PolySync

In true New Yorker form, Katie Cleary leverages her irreverent brand of skepticism into the rigorous application of testing and test automation for autonomous vehicle software & development tools. With the mindset that typical testing methods are insufficient for such a complex domain, she’s leaned into Property-Based Testing as a way to sniff out both the subtle---and sometimes overt---critical bugs in a system before potentially serious problems arise. When she isn't bug hunting, Katie can be found playing scary video games, skating with her roller derby team, or listening to old school Weezer.