Code release process in Facebook
I’ve just come across interesting video in which Chuck Rossi, Core Release Engineer in Facebook with 27 years of experience as an engineer in Silicon Valley, talks about release process in Facebook and how they manage to push updates every day.
If you don’t have time to watch video here’s a list of the most interesting things I’ve noted watching it:
- employees at Facebook are using latest.facebook.com (test branch) so as they are using Facebook they actually testing code before it’s release,
- they use Facebook groups to communicate, one of them is named: “Bugs forum”,
- the whole world is using Facebook so they don’t use production for testing,
- they also use IRC in communication between engineers and IRC bots,
- they have a special tool to track PHP errors and exceptions (with a call stack) where they can see diagrams of frequency of error occurrence and where it occurs (mobile/api/…),
- they have a tool they call “Gate keeper” which can push revs to a different groups of Facebook users, ex. only users located in a single country, only 1% of Facebook users etc.,
- Core Release Engineers can rate programmers (give them stars) so then they can estimate the risk of pushing his/her code,
- they compile PHP code using HipHop, Facebook compiled binary is 1GB and it compiles 10 minutes (!),
- they push that binary across block using BitTorrent, it spreads in 50 minutes.
