Implementing pushState for twitter.com

As part of our continuing effort to improve the performance of twitter.com, we’ve recently implemented pushState. With this change, users experience a perceivable decrease in latency when navigating between sections of twitter.com; in some cases near zero latency, as we’re now caching responses on the client.

This post provides an overview of the pushState API, a summary of our implementation, and details some of the pitfalls and gotchas we experienced along the way.

API Overview

pushState is part of the HTML 5 History API— a set of tools for managing state on the client. The pushState() method enables mapping of a state object to a URL. The address bar is updated to match the specified URL without actually loading the page.

history.pushState([page data], [page title], [page URL])

While the pushState() method is used when navigating forward from A to B, the History API also provides a “popstate” event—used to mange back/forward button navigation. The event’s “state” property maps to the data passed as the first argument to pushState().

If the user presses the back button to return to the initial point from which he/she first navigated via pushState, the “state” property of the “popstate” event will be undefined. To set the state for the initial, full-page load use the replaceState() method. It accepts the same arguments as the pushState() method.

history.replaceState([page data], [page title], [page URL])

The following diagram illustrates how usage of the History API comes together.


Diagram illustrating use of the HTML 5 History API

Progressive Enhancement

Our pushState implementation is a progressive enhancement on top of our previous work, and could be described as Hijax + server-side rendering. By maintaining view logic on the server, we keep the client light, and maintain support for browsers that don’t support pushState with the same URLs. This approach provides the additional benefit of enabling us to disable pushState at any time without jeopardizing any functionality.

On the Server

On the server, we configured each endpoint to return either full-page responses, or a JSON payload containing a partial, server-side rendered view, along with its corresponding JavaScript components. The decision of what response to send is determined by checking the Accept header and looking for “application/json.”

The same views are used to render both types of requests; to support pushState the views format the pieces used for the full-page responses into JSON.

Here are two example responses for the Interactions page to illustrate the point:

pushState response

{
  // Server-rendered HTML for the view
  page: "

",
  // Path to the JavaScript module for the associated view
  module: "app/pages/connect/interactions",
  // Initialization data for the current view
  init_data: {…},
  title: "Twitter / Interactions"
}

Full page response


<b>{{title}}</b>


{{page}}

Client Architecture

Several aspects of our existing client architecture made it particularly easy to enhance twitter.com with pushState.

By contract, our components attach themselves to a single DOM node, listen to events via delegation, fire events on the DOM, and those events are broadcast to other components via DOM event bubbling. This allows our components to be even more loosely coupled—a component doesn’t need a reference to another component in order to listen for its events.

Secondly, all of our components are defined using AMD, enabling the client to make decisions about what components to load.

With this client architecture we implemented pushState by adding two components: one responsible for managing the UI, the other data. Both are attached to the document, listen for events across the entire page, and broadcast events available to all components.

UI Component

  • Manages the decision to pushState URLs by listening for document-wide clicks, and keyboard shortcuts
  • Broadcasts an event to initiate pushState navigation
  • Updates the UI in response to events from the data component

DATA Component

  • Only included if we’re using pushState
  • Manages XHRs and caching of responses
  • Provides eventing around the HTML 5 history API to provide a single interface for UI components

Example pushState() Navigation LifeCycle

  1. The user clicks on link with a specialized class (we choose “js-nav”), the click is caught by the UI component which prevents the default behavior and triggers a custom event to initiate pushState navigation.
  2. The data component listens for that event and…
    1. Writes the current view to cache and, only before initial pushState navigation, calls replaceState() to set the state data for the view
    2. Fetches the JSON payload for the requested URL (either via XHR or from cache)
    3. Update the cache for the URL
    4. Call pushState() to update the URL
    5. Trigger an event indicating the UI should be updated
  3. The UI component resumes control by handling the event from the data component and…
    1. JavaScript components for the current view are torn down (event listeners detached, associated state is cleaned up)
    2. The HTML for the current view is replaced with the new HTML
    3. The script loader only fetches modules not already loaded
    4. The JavaScript components for the current view are initialized
    5. An event is triggered to alert all components that the view is rendered and initialized

Pitfalls, Gotchas, etc.

It’ll come as no surprise to any experienced frontend engineers that the majority of the problems and annoyances with implementing pushState stem from either 1) inconsistencies in browser implementations of the HTML 5 History API, or 2) having to replicate behaviors or functionality you would otherwise get for free with full-page reloads.

Don’t believe the API, title updates are manual

All browsers currently disregard the title attribute passed to the pushState() and replaceState() methods. Any updates to the page title need to be done manually.

popstate Event Inconsistencies

At the time of this writing, WebKit (and only WebKit) fires an extraneous popstate event after initial page load. This appears to be a known bug in WebKit, and is easy to work around by ignoring popstate events if the “state” property is undefined.

State Object Size Limits

Firefox imposes 640KB character limit on the serialized state object passed to pushState(), and will throw an exception if that limit is exceeded. We hit this limit in the early days of our implementation, and moved to storing state in memory. We limit the size of the serialized JSON we cache on the client per URL, and can adjust that number via a server-owned config.

It’s worth noting that due to the aforementioned popstate bug in WebKit, we pass an empty object as the first argument to pushState() to distinguish WebKit’s extraneous popstate events from those triggered in response to back/forward navigation.

Thoughtful State Management Around Caching

The bulk of the work implementing pushState went into designing a simple client framework that would facilitate caching and provide the right events to enable components to both prepare themselves to be cached, and restore themselves from cache. This was solved through a few simple design decisions:

  1. All events that trigger navigation (clicks on links, keyboard shortcuts, and back/forward button presses) are abstracted by the pushState UI component, routed through the same path in the data component, and subsequently fire the same events. This allows the UI to be both cached and handle updates in a uniform way.
  2. The pushState UI component fires events around the rendering of updates: one before the DOM is updated, and another after the update is complete. The former enables UI components such as dialogs and menus to be collapsed in advance of the page being cached; the later enables UI components like timelines to update their timestamps when rendered from cache.
  3. POST & DELETE operations bust the client-side cache.

Re-implementing Browser Functionality

As is often the case, changing the browser’s default behavior in an effort to make the experience faster or simpler for the end-user typically requires more work on behalf of developers and designers. Here are some pieces of browser functionality that we had to re-implement:

  • Managing the position of the scrollbar as the user navigates forward and backward.
  • Preserving context menu functionality when preventing a link’s default click behavior.
  • Accounting for especially fast, indecisive user clicks by ensuring the response you’re rendering is in sync with the last requested URL.
  • Canceling outbound XHRs when the user requests a new page to avoid unnecessary UI updates.
  • Implementing the canonical AJAX spinner, so the user knows the page is loading.

Final Thoughts

Despite the usual browser inconsistencies and other gotchas, we’re pretty happy with the HTML 5 History API. Our implementation has enabled us to deliver the fast initial page rendering times and robustness we associate with traditional, server-side rendered sites and the lightening quick in-app navigation and state changes associate with client-side rendered web applications.

Helpful Resources

—Todd Kloots, Engineer, Web Core team (@todd)