User Analytics: Building Flickr Stats
Flickr’s statistics product was built to give users insight into the ways that people are finding their photos, by providing traffic counts broken down by any combination of photo, date, referring page or search term. To display this information we’re storing “almost real-time” referral counts for all 2 billion photos on Flickr and creating nightly rollup reports for hundreds of thousands of accounts.
This session will examine every aspect of how a four-person team designed and built this system, from initial product development to post launch operational issues. We’ll look at a number of technical implementation details, including techniques for graph rendering and the optimization of MySQL to handle large-scale data collection.
We’ll also discuss how a handful of simple architecture patterns - federation, background tasks, caching and buffering - allow Flickr to quickly create products like this at scale and how you can do the same.
comments
Excellent session! Real world problems with real world solutions. I like how sometimes the solution did not fix the problem or it created another problem....and how he walked us through the whole process of where they were and where they ended up. Excellent talk - slides please?
This was my absolute favorite session of the Expo. It was extremely relevant to what I'm currently doing, and the topics/problems covered are all either things we've had to deal with in the past or things that we hope to have to deal with.
Overall excellent, I walked out of there fired up with some ideas to take home with me. Great job!
This was a great session that provided insights into the kind of real-world problem solving that developers and product managers have to deal with all the time. Paul had great examples of thinking through the design of a feature at large scale while balancing desired functionality with technical trade-offs. I also liked how Paul presented a cohesive story from the first design challenge to the completion of the feature.
Thanks everyone for the kind words. I really enjoyed giving the talk, and I'm glad you all found it useful.
If you're interested, the slide are available as a PDF at http://www.paulhammond.org/2008/10/flickrstats/.
We're planning to put post more details on some of the subjects I covered in my talk at the flickr code blog (code.flickr.com), so it's worth keeping an eye on that.
I think this was a very good session, there was a very practical problem solution focus which was very informative.