The Case for Getting Rid of TestFlight Review
June 12, 2020
I tweeted today about how I think TestFlight review should become a thing of the past and many developers seemed to agree, but some had questions so I wanted to expand on my thoughts a little.
TestFlight’s awesome. But like App Store submissions, TestFlight betas also require a review by Apple. At first blush, such a review sounds sensical. TestFlight can distribute apps to up to 10,000 users. If that were to run completely unchecked you could have potentially mini-App Stores running around with sketchy apps being distributed to lots of people.
But the point I’ll try to make in this article is that the current system TestFlight employs doesn’t do much to prevent this, and further creates a lot of friction for legitimate developers.
The Review Process
For TestFlight, when you submit a new version number, it requires a new review. But new build numbers do not (build numbers are like a secondary ID for a version as it goes through development). For instance, I could push a new version of Apollo to TestFlight, version 1.8 (build number 50) and it would need review, but builds 51, 52, 53, etc. of the same version do not require any review.
The Problem
Do you see the issue here? There’s not really any oversight into what you can change in those new builds. You could completely change your app into something different, upload it under a different build number, and so long as the previous version was approved and you don’t change the version number, you could send the new one out to thousands of people.
Someone looking to distribute, say a console emulator (that Apple doesn’t allow in the App Store), could upload their app as a fun turtle themed calculator app (TurtleCalc™) and get approved on TestFlight, only to update it into that emulator for build 2 and send it out to thousands of people.
As a Developer
On the flip side, for an actual developer with an app on the App Store, it causes a ton of friction, because the other rule of TestFlight is such that once a new version goes live on the App Store, you can’t push any new builds to TestFlight without a new version and starting the review process again.
So if you find a bug in the public version of your app, and want to beta test the fix, you have to wait a day or two for it to be reviewed by Apple before it can even go into beta testing. A 3-lines-of-code bug fix requires re-review, meanwhile, if you’re a bad actor and you just leave the app in TestFlight without ever pushing it to the App Store, you can just update it endlessly without any review whatsoever.
That means as a developer you’re stuck in this gamble of “Should I just release it to the App Store without any testing? It’s just a bug fix after all, what could go wrong?” versus “Should I let it keep crashing and wait for the TestFlight review to occur so I can test this new build first, even if it means crashing for days more?”
In a perfect world, you could push that fix out to testers immediately, validate the fix, then submit it to the App Store.
As a result you have this system that A) doesn’t seem to do anything to stop people submitting nefarious updates but B) introduces a ton of friction to legitimate developers.
“It Serves as an Early Review for the App Store Before Continued Development”
Some argue that it lets you “test the waters” with an app or an update before submitting it to the App Store at large. For instance you have an idea that you’re not sure will get through app review, so you build a quick version of the app, and submit it to TestFlight, and the review will let you know if Apple will approve it.
Unfortunately it doesn’t work like that. Getting through TestFlight review has no bearing on getting through the eventual App Store review. I’ve had builds go through TestFlight review, get the stamp of approval, test it in TestFlight for months, and then when I ultimately submit it the update gets rejected.
TestFlight reviews are not at all an accurate way to gauge what the reviewers will think. It’s far more lax.
It Often Requires Double Review
Even more confusingly, if I decide to take the gamble and just release the bug fix to the App Store and hope all goes well, it’ll goes through a quick review, then it will go live on the App Store.
But if I want the TestFlight users to use that same version that just got approved, they straight up can’t. Even though it went through the more strict public App Store review, the exact same build has to be reviewed separately for TestFlight. This adds a confusing delay for testers (not to mention extra work for Apple) and is very weird.
TestFlight Review Takes Longer than App Store Review
Despite being more lax a review process (as shown above), it takes longer to review. This kinda makes sense, you would hope the majory of staff would be focused on the public App Store review which affects the most users, but it feels bizarre to submit an app to the App Store and TestFlight at the same time (because double review) and the App Store version goes out the same day while the TestFlight version takes a day or two.
This greatly disincentivizes testing builds when the process to actually get them out takes so long.
There’s Already Workarounds
A lot of developers, aware of the above constraints, employ strategies for getting around this process almost completely.
- As soon as you submit the version to the App Store, you can immediately submit the same version plus one (so 1.8.3 on the App Store, 1.8.4 on TestFlight) even without any changes (just a bumped version number), get it through review, and then the next time you need to test a beta build you have an approved version you can start shoveling new builds onto.
- An even more clever method some employ, is to just have an astronomically high version number only for TestFlight. So if your App Store version is 1.8, your TestFlight version is 1,0000. That way your TestFlight build is always ahead of the App Store version, and once that version gets approved the first time, you can indefinitely add new builds onto it. A lot of developers do this, and it’s clever, but I personally fear angering the App Store folks.
You might be asking, “Okay… why not just do one of those methods then?”. And you totally can, but in neither case is the app actually being reviewed, in the first it’s an identical version that’s tweaked “secretly” later, and in the second it’s a single version that gets tweaked forever. This effectively shows how little the review process actually contributes.
Getting Rid of TestFlight Review Could Speed up Normal Review
If TestFlight review were to go away for the reasons outlined above, all the awesome folks on that team could be relocated to the “normal” App Store team, which could see an even faster review process. The review process is so much better now than it has been in the past, typically under a day (it used to be over a week!), but can you imagine submitting a build and it being available within a few hours being the norm? That would be fantastic!
Solution
I think just getting rid of it completely is fair. As shown, the current process does next to nothing to prevent people from distributing questionable builds, and instead is just a pain for legitimate developers.
Is it possible that behind the scenes Apple re-reviews builds and might yank them if they find out they break their rules, say a game console app that’s been getting new builds but no new reviews from Apple for a year? Totally! And I think that’s the system they should simply extend everywhere.
Do away with the review system all together, and have a random review process that occurs after the fact, every so often, perhaps transparently and based on the amount of testers in the beta (a beta with 8,000 users is more dangerous than one with three people).
So you submit your version, it immediately goes out to all testers, and then a little while after Apple might flag it for random review. If it passes, it’s completely transparent to you. If it gets rejected, it’ll be pulled.
End
TestFlight’s great and I love it, but decreasing friction in beta testing would be a massive help.