What happened with the Ambiance 3.0 failure

The last two days were pretty bad for me.  I fucked up bad.  I mean really bad. really really bad.  As a developer you expect bugs, and even though you do your testing and QA, some always slip through the cracks.

 

I was super excited to have an Ambiance 3.0 release on the iOS 4 release date as it would mean extra exposure and press. The update came and I watched my servers as thousands and thousands of customers started updating the app.  Then the emails started trickling in…

 

Some users were reporting of their sounds being lost in the update, others reported not being able to save sounds and other about crashes.  It was only a few so I thought it might be an isolated bug or two.  Then the emails started FLOODING in.  I am talking about hundreds and hundreds of emails, each with varying degrees of intensity.

 

"WHAT HAPPENED TO AMBIANCE I PAID FOR!"

"Was one of my favorite apps"

"FUUUUUUUUCCCCCCCCCCKKKKKKK UUUUUUUUUUUUUUUUUUUUUU"

Ouch.

I immediately pulled the app from the store to stop people from updating to the now clearly-bad update.  At this point, about 15,000 customers got the updated app and I feared the worst.  Of all types of bugs, the last kind of bug you want is a data loss bug and it seemed that this might just be that.  I was able to grab some of my test phones and after trying to duplicate, I could not.  My app worked fine.  All these customers can't be wrong though so I contacted a few of the nicer ones to send me their import file.

Once I got the import files and scanned them for common links, I tried using one of them to import. BAM! I ran into all three major symptoms: Failed Import, No Downloading, Crashes.  Uh oh…

 

BUG #1

After a few minutes sleuthing, I was able to track down bug #1.  Get this: On my test devices, I never rate sounds. This is because I don't want to affect the sound ratings unfairly.  Thus, NONE of my test devices had a sound with a rating in it.  Of course, there was a bug in the import script on the Rating line.  This bug was preventing users from importing their old libraries because of an exception.  That exception was caught and the app went along without the users sounds imported.

 

BUG #2

Bug #1 was bad but it wasn't over. I still couldn't figure out how users were not being able to download sounds.  I tried to take my mind out of developer mode and acted like a random user.  I tried the update process from scratch again, but this time, after the failed import, I did what most users did: tried to use the apps restore functionality to restore using the backup file.

 

This file was created specifically for this purpose and I have no regrets about doing it.  Unfortunately, bug #2 lied in the import database path.  Using the apps restore function with a backup file from Ambiance 2.x had a bug that caused the core data database to be overwritten by the old non Core-Data db.  The restore process would overwrite the DB, then ask the user to restart, then on restart, would open the DB and re-download sounds that should be there.  Except that now the DB can't be opened by Core-Data, and nothing new can be saved either.

FUCK!

The Fix

I patched them both up which took a few hours because I now had to support recovery from at least 4 different potential error states (2^2). I also used the time to do additional checking and failsafes into the restore method to prevent this and other potential failures from happening.  On the bright side, I was able to figure out a way to recover the users data as long as they did not remove the app from the phone.  All the sounds were in the Documents directory still, so I was able to make sense of the old and new DB's,correctly do the import, and correctly transfer the old data.

I uploaded the app to iTunesConnect on Monday night and send of a few emails to request an expedited review. Meanwhile I sent out copies of the update to Beta Testers to see if any issues were still there, ready to pull the app from the review queue if necessary.  Everything went smoothly so I began to wait, fearing the worts with a 2 week review time.  I am not sure if the request was granted or if it was pure luck, but tonight (after 2 days) the update was approved.  1 hour into the availability, about 2000 people have updated and I have yet to hear any issues.  

So what did I learn here? 

- I need to have a more rigorous QA cycle

- Rushing to get the app out on release day can bite you in the ass

- Make sure the "restore" function of your app actually will restore the app from all states.

- Your customer will genuinely hate you if their $2.99 app is not working for 1 minute.

I hope the ratings for Ambiance weren't affected by this hiccup and that these past 2 days can fade into the deep hole of buried developer failures.

The Ambiance 3.0.1 update is now live in the store (http://bit.ly/ambiance)

Was this page helpful for you? Buy me a slice of 🍕 to say thanks!