Open source

Introducing Twitter Image Pipeline iOS framework for open source

By
Wednesday, 1 March 2017

Today, we’re excited to open source our Twitter Image Pipeline iOS framework, or TIP for short.

Why a new framework?

In 2014, Twitter embraced sharing and displaying images in its apps, which was a great enhancement to the communication platform built on brevity. But with that we had accrued a large swath of tech debt, were missing some key features, struggled with reliability, and had many different complex approaches to our images. We took some time to assess where we were at and identify where we wanted to go before building a new framework to solve those pain points. Once we adopted this new iOS framework, we were able to take advantage of its features and have iterated on it over the course of 2 years. Today we’re happy to share it with you so you can enjoy the same benefits.

What does TIP offer?

TIP is a framework that attempts to solve many problems around image loading and caching. To give an overview of what it offers, let’s dig into some of the main parts of TIP.

Caches

A major part of being an efficient pipeline for loading images is the caches. From the very start, we knew we needed a strong caching system. Before TIP, Twitter had some serious problems with its image cache and we needed to have those solved from the start so that once we completed our migration to TIP we could be rid of those issues.

First, we needed to address a user concern around account logout. Since we had a singular caching system, all cached images were shared between all Twitter accounts that were logged into the app. Because Twitter for iOS supports multiple accounts, this presented a problem when an account was logged out. We couldn’t clear images for the logged out account, and we didn’t want to purge the entire cache to lose all the performance it offers. This was a waste of storage and left behind unwanted images in the cache. To fundamentally solve this up front, we elected to silo TIP via “pipelines,” with each pipeline having its own distinct caches which can be individually purged. This use case might not translate to all apps and is completely optional, but it was an important starting point for building TIP.

The caching per pipeline in TIP is separated into 3 caches: the first synchronous access cache is in-memory images that are already rendered to a specific resolution, the second asynchronous cache is in-memory images that aren’t necessarily scaled or even decoded yet, and the last asynchronous cache is an on-disk cache for longer persistence. This offers great performance in loading images while balancing the tradeoffs of how far images are cached. When building these caches, we focused on thread safety and data consistency. Twitter had a long-standing fundamental flaw with our previous caching system as well. There was a race condition which, when triggered, would cause the same image to be loaded from the network twice, and both loads would attempt to write to the same file, resulting in a corrupted image in the cache. Slow networking and larger images exacerbated the chances corruption would occur. TIP does not share this flaw and the issue has been eliminated.

A particularly problematic part of our previous cache was that it was unbounded. The previous design didn’t build in the concept of limiting the cache size and relied solely on TTLs (time-to-live values) to purge images. This led to our image cache reaching 2GB or even larger with no recourse for the user. TIP, however, uses LRU (least-recently-used) mechanism for purging, has configurable limits to how big the cache can get, and has the ability to be cleared (either per pipeline or globally). After a full migration to TIP, Twitter no longer feels the pain of 2GB caches and achieves a better end-user experience with a 128 MB limit of on-disk storage. The TIP framework also allows us to add app data usage settings to clear the cache.

The last need we had when building and adopting TIP was a way to transition from the legacy caches to the new TIP caches. We elected to build support for plugging in additional caches to each image pipeline so that when TIP didn’t have an image, the image could be loaded from the legacy cache before trying the network. This allowed our caches to progressively transition over to TIP without the impact of extra networking from the user.

Fetching Images

The core of an image pipeline is the fetching of a requested image. When building TIP, we had existing use cases that needed to be met and table stakes in performance to achieve, but we also wanted to really build out a robust system for image loading so we could maximize the user experience while minimizing resource impact, particularly with networking resources.

It was noted early on that we have many different sizes of images, a.k.a. variants, that we use in the Twitter app. We were being wasteful having different parts of the app load many different variants. It wasted cache space, cost data, and took time to load new variants over the network. The improvement was to have TIP offer easy to use support for loading existing images that were cached even if they weren’t the correct variant. Variants that are larger than the one being fetched can simply be scaled to the appropriate size, avoiding the network. Variants that are smaller than the one being fetched can be offered as a preview for the user to see while the full image is being loaded, presenting content to the user immediately so they can see something while they wait for the higher quality variant to show up.

We really cared about the network utilization, and wanted TIP to be completely transparent and robust in how it loads images over the network. TIP will automatically coalesce any two (or more) fetches that are for the same image variant, saving on networking. TIP also supports image download resuming; instead of a failed or canceled image load throwing away whatever progress was made, TIP will persist that data so the next time that image is fetched, it can be resumed from where it left off.

A principle we adopted as we started building TIP was to get content to the user as soon as possible. With that in mind, we set out to ensure that TIP had support for progressive loading of images, which gets the content to the user as soon as possible while the image continues to load and gain in fidelity. It proved very successful as a feature, and we’ve since transitioned all Tweet images to be PJPEG. Users with fast internet will see no difference than with JPEG, but the majority of the world that access the internet through a slow connection can see images on screen much faster.

Supporting a diverse set of image codecs was something we learned we needed as we were using TIP. We experimented more and more with different image codecs and ended up making 3 iterations that really benefit TIP. First, we added animated image support (specifically with GIFs). Second, we added support for all ImageIO supported codecs. Last, we abstracted out how TIP encodes and decodes so that codecs are pluggable and any custom codec can easily be added to TIP, either to supplement the existing codecs or replace any of them. With this diverse codec support, Twitter has been able to easily experiment with many image formats including PNG, JPEG, PJPEG, GIF, JPEG-2000, WebP and even custom codecs.

The last critical component to TIP being robust for Twitter’s image fetching needs is the networking it executes with. Twitter has its own network layer abstraction through which all networking goes through, which is necessary for things like robust network performance measurements and control over networking operations. By abstracting out networking, TIP supports any network layer to be plugged in, but for simplicity uses an NSURLSession based option as default.

Beyond fetching images, TIP supports the ability to store images to the caches and is a feature that gives us a lot of flexibility in our user experience. We can now easily support taking the image that is being Tweeted and put it into the cache so that when that Tweet shows in the user’s timeline, the image is already there and doesn’t have to be loaded over the network. To round off control over caches, specific images can be purged too.

Debugging and Observability

Beyond being a framework that provides features, TIP offers detailed insights into what is happening and how things are performing. Consumers of TIP can observe pipeline behavior, be notified of problems that crop up, see detailed logging, and handle robust errors. As an added utility for developers, we built in tools for use at runtime. With TIP’s UIImageView subclass, it is trivial to view debug details of a fetched image with an overlay on the image view. Even more useful is the ability to inspect caches to see what images are cached, including both complete and incomplete loads, with the details of each image. Twitter uses this feature with a simple debug UI for our developers to trivially inspect our caches and debug.

Who is TIP for?

After TIP’s evolution to where it is today, we feel that TIP has enough bang for the buck to be an asset to any app developer or development team that really wants the most control and performance with their images in a single framework.

What about Android?

Great question! Twitter has tried a number of iterations on Android for image loading as well, and currently have adopted Facebook’s Fresco Library. It’s feature rich, reliable, thoroughly documented and well maintained. We couldn’t be happier with using their open source project as an image pipeline for Android.

Where can I find TIP?

As with all Twitter open source projects, the Twitter Image Pipeline framework for iOS project can be found on Twitter’s github page, specifically at https://github.com/twitter/ios-twitter-image-pipeline. We hope you like it!

Be sure to check out our other open source projects, including the Twitter Logging Service framework for iOS (TLS) which can be found at https://github.com/twitter/ios-twitter-logging-service.

This post is unavailable
This post is unavailable.