iOS 8 – A Game Changing Update for the Apple TV

How some behind-the-scenes changes to AirPlay turned the Apple TV into a real game console

If you’re as serious about iOS gaming as I am, chances are you’ve been eagerly awaiting the day Apple releases a high-end Apple TV with an App Store full of games and an official game controller. Due to a variety of factors, mostly related to content deals with cable companies, Apple has thus far been unwilling to release such an Apple TV (they like to make a big splash with the product, and that requires both gaming and television content – can’t do one at a time).

Such an Apple TV product is definitely on the horizon, with various well-connected members of the Apple community mentioning that A-list Apple engineers are hard at work on such a device. However, until such a device is formally released, Apple has pushed users towards a stop-gap solution in the form of a technology called AirPlay.

While this stopgap had issues in the past, Apple has made some serious improvements in iOS 8 that turn AirPlay based gaming into a compelling experience. Read on to learn how.

The History of AirPlay

AirPlay is a brand name that encompasses multiple video output technologies, but the important one to discuss here is one called AirPlay Mirroring. In effect, AirPlay Mirroring beams the contents of your iPhone or iPad’s screen on to your TV. When combined with an MFi controller, this allows you to play iOS games the big screen, without needing to ever take your eyes off the TV and touch your device.

Theoretically, AirPlay Mirroring offers a great solution to people wanting an Apple TV game console. Unfortunately, the reality has been far less pretty. The actual mechanics of beaming a 1-megapixel image from one device to another over Wifi, 60 times per second (iOS and most TVs run at 60fps), without compressing the image until it’s illegible, and without a perceptible delay, has been too tall an order for Apple in the past.

In iOS 7 and below, AirPlay Mirroring runs at a maximum of 30fps, resulting in a somewhat jerky picture. Even being able to do that required significant video compression, resulting in a big-screen picture that often looked like a low-resolution YouTube upload. Unfortunately, this compression also requires a significant amount of processing on the device itself, which results in a noticeable delay between what happens on the iPhone’s screen, and the amount of time it takes to compress, beam, and display the video on the Apple TV – some games are rendered almost unplayable on the TV, since this lag simply doesn’t provide enough time for you to react to the picture you’re seeing. While some games manage to play great in spite of these limitations, for the vast majority, AirPlay has simply not been good enough. But that’s about to change, in a big way.

Enter iOS 8 – AirPlay Reborn

There’s no two ways about this: the difference between iOS 7 AirPlay and iOS 8 Airplay is night and day. In iOS 8, Apple has somehow managed to dramatically improve every single aspect of AirPlay Mirroring. It is a stark, immediately obvious contrast.

First of all, iOS 8 broadcasts AirPlay at a consistent 60 frames per second in compatible apps (more on that later). This means that Apple is shooting a 1-megapixel picture from your device to your Apple TV 60 times per second – twice as much as under iOS 7.

But that’s not all. Apple didn’t just double the framerate, they also dramatically enhanced the quality of the video output. In iOS 7, AirPlay output a relatively hightly compressed video that showed obvious compression artifacts in basically every scenario. It was playable, but certainly noticable. In iOS 8, the compression used is much more intelligent – video quality is dramatically better when less motion is occurring on screen and against simpler backgrounds, and against more complex scenes, the compression occurs less perceptibly. In retro-style games with limited on-screen activity, compression is basically imperceptible. In more complex 3D games, compression exists, but manages to look FAR better than under iOS 7. GTA: San Andreas, for example, looked better being broadcast from my iPhone than it ever did on my PS2.

Complex video compression with a high degree of quality generally comes with a big cost: latency. Without getting into too much detail about the nature of video compression, having a compressor that can see “into the future” can give you dramatically better results. Basically, what that boils down to is, the video being broadcast is always a few frames behind, in order to give the compressor the ability to analyze those frames and create a more fluid picture. The issue for gaming is, if you get too many frames behind, the player starts to notice a delay between their actions and what happens on the screen. This is called input lag. A certain amount of input lag is a part of every TV, but AirPlay video piled a good deal more on top of the amount the TV already had, to the point that many games were unplayable.

In iOS 7, this input lag was very noticeable. Holding your iPhone up to the TV, it was easy to see that the iPhone was displaying a picture almost half a second ahead of the TV. In iOS 8, despite major improvements to video quality, the lag has been reduced. Not eliminated entirely, mind you – such a thing would be impossible – but reduced to the point that a great many games are now playable that weren’t before. This is perhaps the biggest improvement iOS 8 brings. There is a lot further to go before every game is playable comfortably, but the work Apple has done in iOS 8 is commendable.

Unfortunately, iOS 8 brings one pretty major AirPlay flaw that iOS 7 didn’t have, and that serves to make certain games unplayable.

A Stutter Away from Perfection

As great as iOS 8’s AirPlay fundamentals are, one somewhat nasty issue made it into the GM. In certain games, the AirPlay picture exhibits a noticeable stutter every second or so. This basically freezes the picture on screen for 5 frames or so, then throws all those frames up at once. Games with this flaw are generally unplayable, and even after closing the game, the lag remains for a minute or two, before generally correcting itself, only to return if you launch another affected game.

Strangely enough, there is little rhyme or reason as to which games exhibit this flaw. Extremely graphically intense games like Dead Trigger 2 and GTA San Andreas have no flaw at all, whereas the relatively simple Sonic series stutters to the point of unplayability.

I’m still in the early phases of trying to nail down exactly what is causing this stutter issue, and I plan on experimenting with different network configurations and devices to try to determine whether those factors have any impact on it. I’ll update this post when I learn more about this issue, or if Apple manages to fix it with a subsequent update.

The Future

The good news about AirPlay is, it is a technology that can grow and improve along with the components of the iPhone / iPad and the Apple TV. Network technology in these devices today is limited to 802.11n – a fast standard, but one that does provide a bottleneck in maximum bitrate and range. The new iPhone 6 series is being upgraded to the much faster 802.11ac standard, and presumably updates aren’t too far behind for the iPad and the Apple TV.

In addition to wireless upgrades, improvements to the GPUs inside the iPhone and iPad can make a huge difference in broadcast quality. Current GPUs are restricted from working too hard on AirPlay, because if they draw too much power for broadcast, the performance of the game itself starts to suffer. Newer GPUs will include more cores which can be dedicated exclusively towards video processing, and take some of the load off the cores powering the games themselves.

The biggest potential for improvement is in the video encode itself. AirPlay today uses a video compression technology called H264 – a high quality, adaptable, industry-standard codec that can be found in everything from Blu-Rays to web video to teleconferencing. As well as h264 has served us up until today, the industry has been hard at work designing an upgraded codec called h265, which is capable of around 50% greater efficiency – meaning it can cram 50% more quality into the same amount of bandwidth as h264.

Enabling h265 over AirPlay will likely require a new Apple TV with an h265-capable GPU. The good news is, such a chip is easy to find – the A8 chip in Apple’s newly-announced iPhone 6 line is fully capable of encoding and decoding h265 video. When the A8 chip makes its way into the Apple TV, I fully expect an upgraded h265-encoded version of AirPlay will come along for the ride.

But thats all for the future. The fact of the matter is, even today, the vast majority of iOS games are playable on the Apple TV with iOS 8. Apple has done a massive amount of work on this front. The experience of using an iPhone in an MFi controller today, streaming to the Apple TV, is finally a good enough experience that I can recommend it wholeheartedly (at least once the stuttering bug is ironed out).

If you have an iPhone / iPad and an MFi controller, and you want to take your gaming to the TV, Apple finally has you covered with AirPlay and the Apple TV.