Ryan Paul at Ars Technica visited the Facebook HQ and got an exclusive look at Facebook’s release engineering team. His lengthy feature has some fascinating insights into how the company operates and pushes updates. Some excerpts:
the entire Facebook application, is approximately 1.5GB in size.
Rolling out a Facebook update takes an average of 30 minutes—15 minutes to generate the binary executable and another 15 minutes to push the executable
Facebook created its own custom BitTorrent tracker [to distribute the updated binaries to all servers]
Facebook employees who access the social network from within the company’s internal network will always see an experimental build […] When employees want to see the current production version of the website from within the network, they use a separate address.
Facebook hosts its own Internet relay chat (IRC) server for internal collaboration. […] Rossi can send a command to a bot that will attempt to get the developer’s attention through several different communication channels
much of [Rossi’s] his work was done in a browser and in terminal windows. When he was ready to roll out the update, he issued a command in one of the terminals to begin the process.
We watched the status of the rollout in one of Facebook’s Web-based system monitoring tools. The webpage displayed a large progress bar showing the percentage of the company’s servers that had successfully been updated to the new binary.
Any given page request can be handled by any of the servers in Facebook’s infrastructure.
Facebook continues operating at nearly full capacity during an update.
The many data sources tracked by Facebook’s internal monitoring tools even include tweets about Facebook. […]
he’s only had to do it [reverting to a previous version] a few times.
When a developer’s code disrupts the website and necessitates a post-deployment fix, the incident is tracked and factored into Facebook’s assessment of the developer’s job performance.
Facebook’s developers all have a “karma” rating that is tracked through the code review system. Rossi can increase or decrease a developer’s karma by clicking on thumbs-up and thumbs-down icons (like Reddit)
Facebook’s developers are creating their own bytecode format and custom runtime environment, called the HipHop virtual machine […] the company will be able to compile its PHP source into bytecode that will be executed by the virtual machine.
It is a fascinating read!