The Tech Behind the New Twitter.com

The Twitter.com redesign presented an opportunity to make bold changes to the underlying technology of the website. With this in mind, we began implementing a new architecture almost entirely in JavaScript. We put special emphasis on ease of development, extensibility, and performance. Building the application on the client forced us to come up with unique solutions to bring our product to life, a few of which we’d like to highlight in this overview.

API CLIENT

One of the most important architectural changes is that Twitter.com is now a client of our own API. It fetches data from the same endpoints that the mobile site, our apps for iPhone, iPad, Android, and every third-party application use. This shift allowed us to allocate more resources to the API team, generating over 40 patches. In the initial page load and every call from the client, all data is now fetched from a highly optimized JSON fragment cache.

THE JAVASCRIPT API

We built a JavaScript library to access Twitter's REST API for @anywhere which provided a good starting point for development on this project. The JavaScript API provides API fetching and smart client-side caching, both in-memory and using localStorage, allowing us to minimize the number of network requests made while using Twitter.com. For instance, timeline fetches include associated user data for each Tweet. The resulting user objects are proactively cached, so viewing a profile does not require unnecessary fetches of user data.

Another feature of the JavaScript API is that it provides event notifications before and after each API call. This allows components to register interest and respond immediately with appropriate changes to the UI, while letting independent components remain decoupled, even when relying on access to the same data.

PAGE MANAGEMENT

One of the goals with this project was to make page navigation easier and faster. Building on the web’s traditional analogy of interlinked documents, our application uses a page routing system that maintains a strong relationship between a URL and its content. This allows us to provide a rich web application that behaves like a traditional web site. Doing so demanded that we develop a rich routing model on the client. To do so we developed a routing system to switch between stateful pages, driven by the URL hash. As the user navigates, the application caches the visited pages in memory. Although the information on those pages can quickly become stale, we’ve alleviated much of this complexity by making pages subscribe to events from the JavaScript API and keep themselves in sync with the overall application state.

THE RENDERING STACK

In order to support crawlers and users without JavaScript, we needed a rendering system that runs on both server and client. To meet this need, we've built our rendering stack around Mustache, and developed a view object system that generates HTML fragments from API objects. We’ve also extended Mustache to support internationalized string substitution.

Much attention was given to optimizing performance in the DOM. For example, we’ve implemented event delegation across the board, which has enabled a low memory profile without worrying about event attachment. Most of our UI is made out of reusable components, so we've centralized event handling to a few key root nodes. We also minimize repaints by building full HTML structures before they are inserted and attach relevant data in the HTML rendering step, rather than through DOM manipulation.

INLINE MEDIA

One important product feature was embedding third-party content directly on the website whenever tweet links to one of our content partners. For many of these partners, such as Kiva and Vimeo, we rely on the oEmbed standard, making a simple JSON-P request to the content provider's domain and embeds content found in the response. For other media partners, like TwitPic and YouTube, we rely on known embed resources that can be predicted from the URL, which reduces network requests and results in a speedier experience.

OPEN SOURCE

Twitter has always embraced open-source technology, and the new web client continues in this tradition. We used jQuery, Mustache, LABjs, Modernizr, and numerous other open-source scripts and jQuery plugins. We owe a debt of gratitude to the authors of these libraries and many others in the JavaScript community for their awesome efforts in writing open-source JavaScript. We hope that, through continuing innovations in front-end development here at Twitter, we'll be able to give back to the open-source community with some of our own technology.

CONCLUSIONS

With #NewTwitter, we’ve officially adopted JavaScript as a core technology in our organization. This project prompted our first internal JavaScript summit, which represents an ongoing effort to exchange knowledge, refine our craft and discover new ways of developing for the web. We're very excited about the doors this architectural shift will open for us as we continue to invest more deeply in rich client experiences. If you're passionate about JavaScript, application architecture, and Twitter, now is a very exciting time to @JoinTheFlock!