Problem and Scope
Design a web crawler that extracts text data from the web and stores it with the goal of using that text data to train a large language model (LLM) like ChatGPT. The crawler needs to run for only 5 days to collect this training data.
Start by confirming the core user journey, exclusions, success criteria, and the constraints that materially affect the architecture.
Requirements to Clarify
A strong answer should establish scope before choosing components.
Functional requirements:
- Crawl the entire web starting from a set of seed URLs
- Extract text data from websites
- Store the extracted text in a database or storage system
- Follow links recursively to discover new pages
- Handle different types of web content (HTML, text)
Non-functional requirements:
- Fault Tolerance: The system should handle failures gracefully and be able to resume crawling without losing progress
- Politeness: The crawler must respect robots.txt rules and avoid overloading website servers
- Scalability: The system must scale to handle crawling 10 billion web pages
- Efficiency: Complete the crawling process within 5 days
Scale and Capacity
Use the workload to justify storage, partitioning, caching, and reliability decisions. Clarify or challenge these assumptions rather than treating them as unquestionable facts:
- Total Web Pages: Approximately 10 billion
- Average Web Page Size: 2 MB
- Crawling Timeframe: 5 days
- Resources: Unlimited resources within reason (assume the company has significant financial resources)
Architecture Discussion
Walk through the important read and write paths, identify ownership boundaries, and explain how the design behaves when dependencies fail. Cover these areas explicitly:
- URL frontier/queue management system
- Crawler workers
- HTML fetcher and processor
- Text extraction service
- URL extraction service
- Data storage system
- Politeness and rate limiting service
- DNS resolver and cache
- Duplicate detection mechanism
- Fault tolerance and recovery system
- Efficiently crawling billions of pages in limited time
- Handling failures and ensuring fault tolerance
- Respecting website politeness policies (robots.txt)
- Preventing crawler traps and infinite loops
- Managing duplicate content
- Distributing the crawling workload
- Optimizing storage for large amounts of text data
- DNS resolution at scale
- Handling dynamic content rendered with JavaScript
Follow-up Questions
Expect the interviewer to test the consequences of your choices. Practise answering these questions with a concrete decision, its benefit, and its cost:
- How would you ensure your crawler respects robots.txt policies?
- Focus: Assess the candidate's answer to "How would you ensure your crawler respects robots.txt policies" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
- What strategy would you use to detect and handle duplicate content?
- Focus: Assess the candidate's answer to "What strategy would you use to detect and handle duplicate content" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
- How would your system recover from failures?
- Focus: Assess the candidate's answer to "How would your system recover from failures" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
- What mechanisms would you implement to prevent crawler traps?
- Focus: Assess the candidate's answer to "What mechanisms would you implement to prevent crawler traps" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
- How would you optimize DNS resolution for your crawler?
- Focus: Assess the candidate's answer to "How would you optimize DNS resolution for your crawler" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
- What approach would you take for distributing crawling across multiple machines?
- Focus: Assess the candidate's answer to "What approach would you take for distributing crawling across multiple machines" for explicit assumptions, a workable mechanism, failure behavior, and consequential trade-offs.
Evaluation Rubric
MockMe evaluates the answer across the following dimensions. A complete answer should connect claims to requirements and explain consequential trade-offs.
- Requirements and scope (15%): Clarifies and prioritizes the required behavior for Crawl the entire web starting from a set of seed URLs; Extract text data from websites; Store the extracted text in a database or storage system. Establishes the constraints that materially affect Design Web Crawler, including Fault Tolerance: The system should handle failures gracefully and be able to resume crawling without losing progress; Politeness: The crawler must respect robots.txt rules and avoid overloading website servers. Strong evidence includes Separates the critical path from secondary features and resolves ambiguous requirements before choosing components.
- Architecture and interfaces (20%): Presents coherent ownership boundaries and end-to-end flows covering URL frontier/queue management system; Crawler workers; HTML fetcher and processor; Text extraction service. Strong evidence includes Defines interfaces and traces important success, retry, and failure paths across the proposed components.
- Data and scaling (25%): Uses workload assumptions such as Total Web Pages: Approximately 10 billion; Average Web Page Size: 2 MB to justify capacity and partitioning decisions. Explains the data, state, or model strategy for URL frontier/queue management system; Data storage system; DNS resolver and cache. Strong evidence includes Quantifies a dominant workload, identifies the first bottleneck, and explains how the design evolves as that workload grows.
- Reliability, correctness, and safety (20%): Explains concrete failure behavior, recovery, and operational safeguards for Fault tolerance and recovery system; Handling failures and ensuring fault tolerance; Fault Tolerance: The system should handle failures gracefully and be able to resume crawling without losing progress. Strong evidence includes States the required correctness or consistency boundary and covers retries, partial failure, observability, and safe degradation.
- Communication and trade-offs (20%): Drives a structured discussion and compares consequential alternatives for Optimizing storage for large amounts of text data; DNS resolution at scale; Handling dynamic content rendered with JavaScript. Strong evidence includes Makes assumptions explicit, answers the question asked, and explains both the benefit and cost of major decisions.