Pushing to Production on a Friday
TL;DR: Due to too much faith on Apple’s Xcode, I assumed that a simple recompile using a recent update of Xcode, for a minor patch bug fix release of an app I maintain, couldn’t possibly break anything. I was wrong - Xcode had a bug that causes apps running on older iOS versions to crash catastrophically, and thus impacting all app developers worldwide. I missed that, and thus accidentally released a broken app version.
Sit back, and relax. It’s going to be a long read.
If you’ve seen the large number of jokes about programming online, you would know that pushing an update on a Friday, or any day before taking a few days off for that matter, is highly risky. You’re making the bold assumption that your software will function just fine as soon as it’s onto your user’s hands. If one’s lucky, users would never notice anything has changed, aside from any possible UI changes said software may have. However, we are humans, and humans make mistakes. We aren’t perfect robots being able to code a perfect piece of software every single time… well at least at the time of writing, robots can’t write software yet. Bugs will slip by, regardless of how much QA and testing you’ve done with software. If it’s a serious issue, congrats! Anger is what users go by, contacting support, writing reviews till you can’t leave your chair.
Well, as both a high school student and an indie app developer, this still applies, but in a different way. What are weekdays to the real world are my “weekends” – days not needing to do app maintenance, and which I’ve not a lot of time to respond to support emails. It’s school hours for almost the whole day after all, and I don’t want to mess up in the middle of the day, bad things will happen. Conversely, weekends in the real world are my “weekdays” – days which I can easily do software maintenance and reply to support emails, even as little as 10 minutes if I’m already on the computer. This means that I’m effectively busy for almost 24/7, mostly studying during the weekdays and maintaining and acting as customer support for my apps during the weekends.
From late October till late November, I have a large number of examinations to sit for (they’re not directly related thus different dates), having breaks for about a week every now and then. Personal Diary version 2.5.1 was scheduled to be released on the 7th of November, with about 5 builds being released for beta testing in total beforehand. That’s already a lot for me to handle, considering I also need to go to school. Bonus fact, the scheduled release is one day immediately after an examination I have, reason being that having it after an exam paper means I can probably cope with the possibility that I messed up the update. The app was submitted, reviewed and approved by Apple for release on the 3rd of November – a Sunday. It was effectively a “Friday” for me, as I’ve to attend school lessons the next day. And by the topic of this blog post, you can probably see where this is going.
7 November rolls along, and the automated scheduled release is doing its job starting at 8 AM. Of course, I didn’t bother about it since the 3rd of November, as I’ve done a lot of beta testing for this particular release. The day goes on. First email of the day came in: it’s an empty one, but with automatically attached diagnostics information. I didn’t bother to check at the time, as empty support emails tend to be received as some users think sending an empty email can somehow allow me to know what issue they’re facing. No, I’m not going to go through the log history to figure out what has happened. Next email is where the first warning strikes: “The app keeps crashing, I’ve tried restarted the phone and the app and that didn’t help! Please fix it now, thank you!” – a user requesting a fix asap, paraphrased. While the user didn’t explicitly state when the app crashes, the attached diagnostics logs indicate that the crash occurs whenever creating a new entry, or viewing an entry in the past from the Timeline view. Next up, an hour later, Crashlytics sent an email to me regarding a drop of crash-free users – 94.5% overall, down from about 99.8%.
What could possibly go wrong? My app works just fine on all of my devices, and Apple didn’t reject my app for a buggy release (if you didn’t know, they actually do reject apps for crashing on launch, or not being able to perform the task it claims to be able to do, whether due to a bug or intentionally). Since there are about 1.5K unique active Personal Diary users daily (according to Firebase at least), an approximately 5% drop within less than 5 hours of a release is very serious, especially when I’m just an individual behind the app, not a fully-fledged company capable of easily handling the situation. Note that most people have auto-updates turned on, so many wouldn’t even know an update has occurred.
As soon as I discovered the email (coincidentally school ended and it was lunch time), I quickly fired up a browser on my phone to analyse the crash report on Firebase. Speaking of that, Firebase doesn’t actually work well on a mobile phone, so I needed to request desktop site in order to use the site in the first place. A quick glance at the most frequent crash reports reveals the error.
“Could not instantiate class named _UITextLayoutView because no class named _UITextLayoutView was found; the class needs to be defined in source code or linked in from a library (ensure the class is part of the correct target)”
Well that’s weird isn’t it? _UITextLayoutView looks like an internal class, presumably it’s part of UITextView, which is indeed used in the page for writing your diary entries. I mean, that’s the most suitable built-in text view for this scenario anyway. If that is causing an error, it’s going to be significantly harder to debug. Oh well, I’ll deal with this after lunch.
As I reach for my computer in my room (I didn’t think of taking that to school at the time), I fired up the exact Firebase crash summary link on my computer by handing off from my phone, and analysed the stack trace a little and also what user interactions have led to this tragedy to occur. After identifying that the issue was indeed caused by something regarding any part of the app that loads an embedded UITextView and not knowing what to do next, I copied the error message like any sane programmer would, pasted it in a search engine with quotes in the hopes that a web search can reveal some exact matches of the error message.
Surprisingly, there are results, and are relatively recent ones too. The title of the first result wasn’t about the text view though, even though it contains the error message. It mentions “After upgrading from Xcode 11.1 to Xcode 11.2, the app crashes due to _UITextLayoutView”. Curious, I opened up the StackOverflow link, and indeed confirmed that the issue was not due to my code, but caused by Xcode. Not only that, this issue only occurs on devices running below iOS 13.2, i.e. the app’s flawless if ran on a newer iOS version.
This explains everything – I didn’t experience any issues as I’ve fully updated most of my personal and test devices on hand (which is not a lot) to said iOS version. Furthermore, it appears that I just so happened to have updated to Xcode 11.2 and built my originally fully-functional-app with it, without even realising that it doesn’t work on older devices. With my low number of active beta testers, I believe most of them are also on iOS 13.2 or even iOS 13.3 beta, as they’re likely always on the bleeding edge of things. At least it’ll be an easy fix!
While I’m at this moment of realisation that the issue isn’t my code, I looked at Firebase stats and noticed that less than a quarter of my users are on iOS 13.2 or later. That’s a problem – it means eventually most users will start encountering the issue when trying to write their entry in Personal Diary. Emails trickle in at a fast pace, even with only a handful of users active in any given time, I receive a large influx of emails – overwhelming my capacity to deal with them. I – a one man team – can’t deal with customer support for such a large number of users, I’m likely to burnout. Internet connection is intermittent that day, and I was only able to download the Xcode 11.2.1 GM seed (which supposedly fixes the issue based on the StackOverflow post) after a long 5 hours. All I did was recompiled the app and uploaded it to the App Store, plus a minor bug fix along the way.
While we’re at that, a large banner notice has been put up on my site, and the App Store description regarding the incompatibility with iOS version older than iOS 13.2. I mean, it worked for some people – at least 2 out of the hundreds of emails that came in noticed the message and updated iOS on their own. I’m still amazed that people paid attention, I was expecting it to be entirely ignored.
Version 2.5.2 was submitted for review on 8 PM – within 12 hours of the previous release. It was eventually accepted by Apple in a few hours and pushed to the public via Automatic Updates. You don’t usually see app updates in less than a day, do you? However, I soon realised that this mechanism for auto-updating apps that Apple has isn’t really live – the previous build is still being served to users. If anything, they’re still being updated to the older version 2.5.1 from version 2.5, while the bug fix in 2.5.2 is already available for manual download. This is a problem, as over the course of the first day, almost 100+ emails have been received (and promptly responded to), and the second day sees no slow downs, but in fact an increase in support tickets. I was flabbergasted. I need to deal with so many emails, I just gave up and sent copies of the same message. They’re slightly personalised to the receiving user by mentioning and greeting them with their name though. While I’m busy sending replies to the hundreds of users that sent in reports, new ones still arrive. I could barely handle that, especially when things don’t seem to be slowing down.
This went on for about three whole days before everything cooled down. Not just the crashes, the emails as well. It’s even more infuriating to think about when the latest update has already resolved the issue in less than 24 hours, but you still have to deal with it for such a long time.
Even though majority of users never responded to my email, for those who did, they are happy for the swift response. Some are confused about why they had “just updated this morning” but there’s a new update again, but either way, they’re pleased. If anything, you’d be able to see on the App Store right now that Personal Diary Version 2.5.1 and 2.5.2 are effectively released on the same day. My Personal Diary downloads however was somewhat affected, presumably due to ratings and ranking drops. It has left a scar in my nearly perfect crash-free users stats. I’m lucky that I didn’t get a lot of negative reviews as I’ve an automated prompt after an app crash to contact me for help. Well, except for this one person, who was not only mad, they were also mad that Safe Mode (a functionality within the app to attempt to automatically log and fix issues) didn’t work. Well, Safe Mode isn’t magic anyway, it can’t fix everything.
The only thing that I’m not really comfortable with is the above comment about whether I’m ensuring the app is stable before release. Well, for one, the app is indeed stable, it’s just that I didn’t expect a compiler update to completely break support for older OS versions… that doesn’t happen out of nowhere unless it’s explicitly stated in the changelog. It is definitely partly my fault for not testing it on older versions, but what about Apple – how did they release a copy of Xcode which causes apps compiled with it to be crippled on older devices? You see, not a lot of people want to beta test apps. Probably neither does Apple have lots of beta testers for minor releases like that. Furthermore, beta testers often are more careful and tend to be more technologically literate, and that the beta testers generally are up to date in their software. Maybe that’s why the bug slipped all the way onto users of my app. This is something that’s generally overlooked, including by me.
I’ve responded to the review via the App Store, and in typical “end-user fashion”, they don’t respond back when something is working as intended. I guess that’s how life is – no one will praise when things in life are great for them, but they’ll complain like it’s the end of the world when something doesn’t go as planned.
Regardless, this was an interesting and somewhat traumatising experience, and of course I’m never going to be doing that again. I now take precautions such as using the Phased Release feature of App Store Connect to slowly release an update over a 7 day period via automatic updates, rather than an instant “push”. Hey, at least a lot of people were satisfied that at least I responded to their emails, so that’s something positive I can take away. Hopefully, a similar fault won’t occur again within the lifetime of Personal Diary, or any app I made or will ever make in general.