Flickr’s statistics product was built to give users insight into the ways that people are finding their photos, by providing traffic counts broken down by any combination of photo, date, referring page or search term. To display this information we’re storing “almost real-time” referral counts for all 2 billion photos on Flickr and creating nightly rollup reports for hundreds of thousands of accounts.

This session will examine every aspect of how a four-person team designed and built this system, from initial product development to post launch operational issues. We’ll look at a number of technical implementation details, including techniques for graph rendering and the optimization of MySQL to handle large-scale data collection.

We’ll also discuss how a handful of simple architecture patterns - federation, background tasks, caching and buffering - allow Flickr to quickly create products like this at scale and how you can do the same.