Nate Weiner

Trust me, I know how badly the App Store's review process can burn you (Ahem).   In fact, while I'm on the verge of trying to push out Read It Later 2.0 as the busier holiday season approaches, I'm expecting even more frustration.

Since Joe Hewitt quit the iPhone platform, a strong discussion has emerged and a lot of good ideas have been surfaced.  However, one bad idea that both Apple and I know will never happen is: ditching the review process all together.

Over the last few weeks, people have talked a lot about how the review process is bad for the open web.  What I think is interesting about this concept is that the one company I believe is probably the most open of all, Mozilla, has a review process for extensions.  All of those free, open extensions you have in your browser?  Those go through a far more rigorous review process, have their actual source code looked over, and sit in a line generally longer than the one at Apple.  Yet, even though this has been happening for years, I would bet no one has heard of any controversies there.

I know this because of my time building add-ons but also because around the time of Firefox's 3.0 release, I was an AMO Editor (extension reviewer).  (Technically, I still am, but have simply not had the time to volunteer recently.)

The review process is absolutely necessary.  Mozilla's reviews stop a LOT of junk from getting through to the end user.  When you browse the AMO directory, you can be assured that every add-on in there is useful to some purpose and more importantly, it's safe.  It's not going to steal personal data, it's not going to install additional programs or do anything not clearly described in the description.

Mozilla's process, though longer and more in depth, differs from Apple's in only a few ways.  Yet, these make all the difference between controversy and satisfied silence.

1. Experimental Add-ons

Anyone can submit an add-on to AMO (Mozilla's add-on directory), it will appear immediately and be available for download.  However, this add-on is flagged as 'experimental' and when the user views/tries to install it, it warns them of the potential consequences of using untested software.

After the add-on has been submitted it can be nominated for public status.  There are a number of criteria your add-on must meet in order to be considered.  Overall your add-on has to be useful to a wide portion of users, well tested, and has been received favorably from those who have tried the experimental version.

Once it's been nominated, it enters the review queue for an AMO editor to look over and eventually reject/approve it.

This means that anyone can get into the directory immediately and start having people try their add-on.  Once it's flushed out further, they can push it through to the more public, much more lucrative (download wise) public side.

If it's an add-on that is only useful to a small number of users, it will stay as experimental and won't clutter up the main AMO directory.

2. Responsiveness

If you send an email to the AMO editors, you can almost always expect a response within a day.  I am still on the email list for the editors address and from time to time check in, but I always see the editors responding quickly and generally with helpful information to answer the developers question.  Often there is thoughtful conversation between the editors trying to determine the proper response before replying.

This is the most important factor in a successful review system. A developer should be able to get any and all questions answered in satisfactory way as quickly as possible.

In the way that Apple works, it's generally pretty difficult to illicit much response from their blackbox.  In the past year I've sent several emails over to Apple's side and was often met with silence or in one case a month long delay before a response. This is unacceptable.

I will say however, the most recent email I sent to the review team was replied to within 17 minutes.  Though the response was short/vague it may mean things have certainly improved since earlier this year.

3. Quick Modifications to Rejections

When an add-on is rejected, you receive an email with the reason why and the name of the reviewer.  Generally, if the reason for rejection was something that can be fixed relatively quickly, you can email that reviewer back, say you resubmitted the fix and they'll take another look at it. You won't have to restart the lengthy wait to be reviewed for something that took you 10 minutes to correct.

This is one of the loudest complaints you'll hear from iPhone developers.  If you wait in line for 2 weeks and then ultimately get rejected for something as minor as an icon, it's incredibly frustrating to have to start all the way back at the beginning of the queue.  The 2 week long wait will seem a lot less unbearable if you know you can still get approved given a minor oversight (or problem with policy).

...

Mozilla's system is far from perfect.  It also suffers from one of Apple's biggest problem in that updates to existing add-ons can take a while to get through the queue.  However, the responsiveness offered by AMO's team, means that if you are rejected for something arguable or minor, that you can still get through just after a few emails and a few changes to your code.

Apple's current process is running on ground that cannot stand for much longer.  The number of apps is growing rapidly, and unless they hire hundreds of reviewers, it's unlikely that all of these applications are getting thorough testing.  From what we, as developers, have seen is that the review generally lasts a minute or two.  From stat tracking to server logs, a lot of developers can see that the reviewer doesn't even use all of the functions within the app.  If this is the case, then the usefulness/stability of the apps appearing in the store are a lot less credible.

They have over 100,000 apps in the store, but let's be honest, a huge number of those are crap.  It would be far more useful to the end user to have an app store that had less apps, but higher quality than vice versa.  By implementing some sort of intermediary (like Mozilla's experimental stage), they could offer developers a place to stand on their platform while being able to dedicate more time to those in the main app review process.  More time means, better reviews, more responsiveness, and less of a wait time.

The App Store review process certainly could use some improvement, but we need to accept that it's there for good and push harder for smarter changes to improve the review process, rather than waste our time trying to abolish it.