Problem and Scope
Design a web search engine that allows users to search for relevant websites across the internet, similar to Google.
Start by confirming the core user journey, exclusions, success criteria, and the constraints that materially affect the architecture.
Requirements to Clarify
A strong answer should establish scope before choosing components.
Functional requirements:
- Users should be able to enter search queries
- The system should find websites relevant to the search query
- Users should see a list of search results with titles, descriptions, and URLs
- Results should be ranked by relevance to the query
- The system should handle pagination of search results
- The system should crawl and index web pages from across the internet
Non-functional requirements:
- High availability: Users expect search to be available 24/7
- Low latency: Search results should be returned quickly (typically < 500ms)
- Scalability: The system should handle billions of web pages and millions of queries per day
- Freshness: The index should be updated regularly to reflect changes in websites
- Efficiency: The crawling process should be polite and not overload target websites
Scale and Capacity
Use the workload to justify storage, partitioning, caching, and reliability decisions. Clarify or challenge these assumptions rather than treating them as unquestionable facts:
- Support indexing of approximately 100 billion web pages
- Handle around 10 billion search queries per day
- Average web page size is about 2MB
- Only 100 billion out of 1 trillion pages on the internet are unique
- Each web page should be crawled, on average, every 10 days
Architecture Discussion
Walk through the important read and write paths, identify ownership boundaries, and explain how the design behaves when dependencies fail. Cover these areas explicitly:
- Web crawler to discover and download web pages
- URL frontier to manage URLs that need to be crawled
- Page processor/parser to extract content and links
- Indexer to create searchable indexes
- Query processor to handle user searches
- Ranking system to determine result relevance
- Storage system for web page content and metadata
- Caching layer for frequently accessed results
- Building an efficient crawler that can handle the scale of the internet
- Designing a politeness policy so websites aren't overloaded
- Creating an index structure that allows for fast retrieval
- Implementing an effective ranking algorithm
- Handling duplicate and near-duplicate content
- Managing the massive storage requirements
- Ensuring freshness of the index
- Scaling the query processing for high-volume searches
Follow-up Questions
Expect the interviewer to test the consequences of your choices. Practise answering these questions with a concrete decision, its benefit, and its cost:
- How would you handle the massive storage requirements for all web pages?
- Focus: Assess the candidate's answer to "How would you handle the massive storage requirements for all web pages" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
- What approach would you take to ensure politeness in your crawler design?
- Focus: Assess the candidate's answer to "What approach would you take to ensure politeness in your crawler design" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
- How would you handle the URL frontier to manage billions of URLs efficiently?
- Focus: Assess the candidate's answer to "How would you handle the URL frontier to manage billions of URLs efficiently" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
- What strategies would you use to detect and handle duplicate content?
- Focus: Assess the candidate's answer to "What strategies would you use to detect and handle duplicate content" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
- How would your system handle the trade-off between freshness and crawling efficiency?
- Focus: Assess the candidate's answer to "How would your system handle the trade-off between freshness and crawling efficiency" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
- Can you explain more about how your indexing strategy would support efficient queries?
- Focus: Assess the candidate's answer to "Can you explain more about how your indexing strategy would support efficient queries" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
Evaluation Rubric
MockMe evaluates the answer across the following dimensions. A complete answer should connect claims to requirements and explain consequential trade-offs.
- Requirements and scope (15%): Clarifies and prioritizes the required behavior for Users should be able to enter search queries; The system should find websites relevant to the search query; Users should see a list of search results with titles, descriptions, and URLs. Establishes the constraints that materially affect Design Google Search, including High availability: Users expect search to be available 24/7; Low latency: Search results should be returned quickly (typically < 500ms). Strong evidence includes Separates the critical path from secondary features and resolves ambiguous requirements before choosing components.
- Architecture and interfaces (20%): Presents coherent ownership boundaries and end-to-end flows covering Web crawler to discover and download web pages; URL frontier to manage URLs that need to be crawled; Page processor/parser to extract content and links; Indexer to create searchable indexes. Strong evidence includes Defines interfaces and traces important success, retry, and failure paths across the proposed components.
- Data and scaling (25%): Uses workload assumptions such as Support indexing of approximately 100 billion web pages; Handle around 10 billion search queries per day to justify capacity and partitioning decisions. Explains the data, state, or model strategy for Indexer to create searchable indexes; Query processor to handle user searches; Ranking system to determine result relevance. Strong evidence includes Quantifies a dominant workload, identifies the first bottleneck, and explains how the design evolves as that workload grows.
- Reliability, correctness, and safety (20%): Explains concrete failure behavior, recovery, and operational safeguards for High availability: Users expect search to be available 24/7. Strong evidence includes States the required correctness or consistency boundary and covers retries, partial failure, observability, and safe degradation.
- Communication and trade-offs (20%): Drives a structured discussion and compares consequential alternatives for Managing the massive storage requirements; Ensuring freshness of the index; Scaling the query processing for high-volume searches. Strong evidence includes Makes assumptions explicit, answers the question asked, and explains both the benefit and cost of major decisions.