Brief Summary
Mark Zuckerberg's guest lecture at Harvard CS50 covers the early architectural decisions and technical challenges faced while scaling Facebook. He discusses database distribution, caching strategies, and the importance of hiring smart people. Zuckerberg also touches on privacy considerations, feature development, and the balance between technical innovation and social impact.
- Early architectural decisions for scaling Facebook
- Technical challenges and solutions for performance
- Importance of hiring intelligent and adaptable people
- Balancing privacy with user experience
- Iterative development and feature evolution
Introduction
Michael D. Smith introduces Mark Zuckerberg, the founder of Facebook, as a guest speaker to discuss computer science in the real world. Zuckerberg's social networking program is used at over 2,000 schools worldwide. Smith welcomes Zuckerberg to share his background and experiences in computer science.
Course Impact and Facebook's Early Days
Zuckerberg outlines his plan to discuss how his Harvard courses influenced the decisions he made while developing Facebook. He hopes to demonstrate the practical value of studying computer science and engineering at Harvard. He mentions starting with course 121 and regretting not taking CS50, suggesting Dustin Moskovitz, his roommate who completed CS50, would have been a better speaker. The initial version of Facebook was written in PHP, which he learned quickly due to his C background.
Scaling Facebook: Architecture and Data Distribution
Zuckerberg discusses the first major decision in expanding Facebook's architecture from a single school to multiple schools. This involved product decisions and privacy considerations. A key decision was how to distribute the data to manage the complexity of computing connections between people. He explains the complexity using big O notation, highlighting the challenge of computing paths between friends, which could become exponentially difficult. To address this, they split the databases, creating one instance of MySQL database for each school, which significantly reduced the computational load.
Infrastructure and Performance Optimisation
Zuckerberg explains that Facebook initially ran on a single rented computer, hosting both the database and the web server (Apache). Distributing the databases allowed them to add more machines linearly as the site grew. However, after reaching 30 to 50 schools, they needed to optimise MySQL and Apache performance. They separated the web servers from the database servers, creating a pool of Apache web servers for load balancing. This setup allowed them to handle varying usage levels across different schools, such as Penn State with 50,000 users versus smaller schools with fewer users.
Caching and Performance Bottlenecks
Zuckerberg discusses encountering performance bottlenecks as traffic increased, particularly with MySQL. A typical query might take two to four milliseconds, but with 100 billion page views a day, each involving multiple queries, this became problematic. They developed a caching layer using Memcache to improve access times to 0.3 to 0.5 milliseconds. However, Memcache had distribution issues and lacked redundancy, causing problems when boxes went down. Eventually, they outgrew Memcache and MySQL indices, requiring them to build extra redundancy on top of these systems.
Q&A: Competition and Technical Advantages
In a Q&A segment, Zuckerberg addresses a question about competition from larger players like Google. He notes that individuals are now more empowered in technology. He highlights that Facebook, with far fewer machines and employees than Google, handles more page views due to architectural advancements. He contrasts this with earlier companies like eBay, which relied on expensive hardware. Zuckerberg emphasises that the ability to rent machines cheaply has enabled smaller teams to achieve significant scale.
Q&A: Open Source and Search Server
An audience member asks about extending Memcache and the challenges of finding good DHT libraries. Zuckerberg explains that they built much of their infrastructure themselves and considered open-sourcing a search server developed by one of their engineers. However, the complexities of licensing and support made it too difficult. He mentions that the search server used a structure other than a hash table, developed by Andrew McCollum.
Q&A: Hiring and Company Focus
Zuckerberg states that he spends most of his time hiring people, emphasising the importance of intelligent individuals as technology becomes more accessible. He believes that smart people are more leveraged to do more things due to technological advancements. Besides hiring, he focuses on designing new features and avoids corporate bureaucracy. He advises that starting a company should stem from a desire to create something cool, rather than just to make a company.
Q&A: Legal Advice and Company Origins
Zuckerberg admits that they didn't consult lawyers early on, which caused some issues later. While he acknowledges the value of legal advice, he prioritised making stuff. He notes that many successful companies, like Google, Yahoo, and eBay, started as passion projects rather than calculated business ventures. He advises that it's better to take action and address issues later than to be overly cautious and get nothing done.
Q&A: Company Evolution and Core Ideas
Zuckerberg discusses how companies evolve beyond their initial ideas, using Yahoo and Google as examples. He explains that Facebook started with the simple idea of finding information about people. He stresses the importance of surrounding oneself with smart people, as the founders' initial understanding doesn't guarantee expertise in all areas. He mentions that the photo features were inspired by existing applications like Flickr, but the ability to tag photos and link them to profiles was unique to Facebook's context.
Q&A: Technology Adoption and Photo Storage
Zuckerberg explains that technology decisions are guided by the expertise of smart people within the company. He describes the process of choosing technologies for photo storage, initially considering large storage solutions like NetApp. They opted for a distributed approach with smaller boxes, RAM, and caching. They then addressed network issues by building a Java applet and ActiveX control for client-side compression. They later used edge caching (like Akamai) to distribute photos closer to users.
Q&A: Software Engineering Methodology
Zuckerberg describes Facebook's software engineering methodology as a meritocracy, where the best solutions and implementations lead to influence. New hires are paired with experienced engineers to learn the company's style and methods. He emphasises that the process is iterative and doesn't require perfection from the start. He highlights the importance of getting the architecture right, as implementation details can be adjusted later.
Q&A: Learning and Hiring Philosophy
Zuckerberg explains that when facing knowledge gaps, they rely on the internet and avoid hiring based solely on skills. They prefer hiring for raw intelligence, believing that intelligent individuals can learn quickly. He cites his roommate Dustin, an economics major, as an example of someone who quickly picked up the necessary skills. He also mentions that EE and math majors often adapt well to computer science roles.
Q&A: Hiring Priorities and Business Skills
Zuckerberg states that he doesn't hire people solely for business skills. He notes that core computer science knowledge, particularly understanding complexity and scale, is valuable in business. He believes that the same intelligence can solve both technical and business problems. He emphasises that they are always building infrastructure and focusing on long-term value rather than short-term profits.
Q&A: Traffic Patterns and Data Retention
Zuckerberg discusses Facebook's traffic patterns, noting peaks around 9:00 PM Pacific time. He mentions that they monitor traffic graphs and react to any blips. Currently, they do not keep a cache of deleted profile information but may consider it in the future.
Q&A: Privacy and Security Concerns
Zuckerberg addresses privacy and security concerns, stating that Facebook's value lies in providing information to the people who the user wants to have access to it. He explains that limiting access to profiles to people within the same school was a core decision. He also emphasises that users have complete control over what information is displayed. He acknowledges the difficulty of controlling what users do with information once they have access to it.
Q&A: Feature Evolution and Design Changes
Zuckerberg discusses the evolution of the Wall feature, which he initially threw together quickly. He explains that changes were made because the original design didn't work as well as intended. The goal was to create a wiki-type thing on profiles, but the implementation had issues. They redesigned it to show the picture and name of the person who posted everything, making it more user-friendly.
Q&A: Facebook's Origins and SMS Integration
Zuckerberg reiterates that the idea for Facebook came from wanting to create a directory where people could find information about others. He explains that the SMS gateways also have an email counterpart, allowing free text messaging. They are in the process of setting up direct text messaging to avoid relying on email gateways.
Q&A: Competition and Data Scraping Prevention
Zuckerberg states that they are not competing with Myspace and that it's a different type of application. He explains that school emails are displayed as images to prevent data scraping. They have implemented various measures to prevent people from aggregating information off of Facebook, including limiting access to profiles at other schools and detecting abnormal viewing activity.
Q&A: Data Usage and Aggregate Statistics
Zuckerberg mentions that they use user data to target posters and are sensitive to privacy concerns. They plan to release aggregate statistics that they think are interesting.
Q&A: Procrastination and Future Features
Zuckerberg admits to procrastinating on Facebook. He reveals upcoming features, including aggregated statistics and a system for clarifying relationships between people. He explains that the goal is to allow users to express how close they are to others in an unbiased way, using bi-directional, factual statements like "I took CS50 with this person."
Q&A: Relationship Analysis and Feature Development
Zuckerberg explains that the new relationship feature will combine user input with event dates to provide a more accurate representation of relationships. He emphasises that Facebook won't rate the relationships but will provide information that users can interpret. He also discusses how new ideas are developed, noting that Facebook has a unique platform for building certain features.
Q&A: Internal Idea Generation and Feature Approval
Zuckerberg describes how employees contribute ideas, often experimenting with the code base. He mentions holding "CEO office hours" where people can show him their projects. He shares an example of a feature that highlights changes in a friend's profile, which he found cool but not necessarily ready for implementation. He emphasises that ideas come from both the ground up and from his own vision.
Q&A: Customisation and High School Considerations
Zuckerberg explains why he doesn't want to allow users to customise the look of their pages, as it would disrupt the directory-like structure of Facebook. He discusses the success of the high school version of Facebook, noting that the purpose is the same as the college version: to allow people to find information about others. He mentions that growth has been good, with more than 5,000 people joining daily.
Q&A: Facebook's Intentions and Skill Recommendations
Zuckerberg states that he didn't intend for Facebook to become a full-fledged business. He remembers arguing with his parents about the value of creating a directory of everyone. He recommends taking the hardest courses possible to challenge oneself. He highlights the value of courses like 161, 121, and 124, which taught him about data structures, algorithms, and the importance of efficient frameworks.
Q&A: Feature Stability and Idea Implementation
Zuckerberg explains that while employees can create whatever they want, not everything makes it onto the site. He personally reviews features before they go live. He reiterates that ideas often come from the ground up, but the final implementation may differ from the original concept.
Q&A: High School Privacy and Social Solutions
Zuckerberg addresses concerns about privacy on the high school version of Facebook, where users are younger. He explains that they rely on social pressure, such as requiring real names and school email addresses, to discourage inappropriate content. He notes that they removed parties from the high school version and deemphasised contact information.
Conclusion
The session concludes with an invitation for attendees to approach Mark Zuckerberg with further questions.
