Mission #: 001

Codename: Operation Dub-Sub

Situation: System Bug + Usability Flaw + Impatient Customers = 😬🤦🏻‍♂️


Summary

For several months we were seeing a steady stream of customer support tickets come through with a common theme: customers were hitting their wash subscription limits sooner than they should be. After digging into the data, we could see that these customers were pulling 2 wash codes from their subscription when they were only intending to pull one.

<aside> 📌 Internally, we started referring to this issue as the Dub-Sub Problem (the double subscription code request problem) — "Dub-Sub" rolled off the tongue a little nicer...

</aside>

This issue stumped our engineering team for quite some time, but eventually we were able to fully resolve the issue. In the end, the final solution was the culmination of several system fixes and usability improvements.

Investigation

Typically, when we receive a customer support ticket that requires input from our engineering team, the first thing we do is look at the customer's data. In many cases, there's an obvious issue that can be found by a trained eye and easily resolved (resolution might be a band-aid fix at first, then a fix to the root issue would be scheduled in at a later time).

So, that's what we did — whenever we received a support request of this nature, we took a look at the transaction data for that customer. Over the course of a few weeks, we noticed the following trends:

  1. Two requests to pull a wash code within milliseconds of each other
  2. Two requests to pull a wash code several seconds apart (anywhere from 10s - 60s)

Trend Analysis

This information certainly helped us make some initial conclusions about what was happening, but we didn't get much closer to a fully understanding of the issue or the potential solution. The 2 trends we observed told conflicting stories...

Trend 1

When we investigated requests that were milliseconds apart, we quickly concluded that this had to be a system issue. There's no way a user could go through the code request flow in the app multiple times in under 5 seconds, let alone milliseconds.

The problem was, our mobile app team claimed it had to be an issue on the API side, and the core team claimed it had to be an issue in the mobile app... Unfortunately, neither team could uncover enough evidence to confidentently point one way or the other, so more investigation was needed...

Trend 2